From 38321844350d9595968dc8613514749fdd2e483a Mon Sep 17 00:00:00 2001 From: Yang Zaizhou <91008302+Mxfg-incense@users.noreply.github.com> Date: Wed, 16 Apr 2025 01:32:50 +0800 Subject: [PATCH 01/63] [flang][docs] Fix typo in array description --- flang/docs/FortranForCProgrammers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flang/docs/FortranForCProgrammers.md b/flang/docs/FortranForCProgrammers.md index 50c83ed7e9bfe..135e6b71711d3 100644 --- a/flang/docs/FortranForCProgrammers.md +++ b/flang/docs/FortranForCProgrammers.md @@ -127,7 +127,7 @@ where type is not necessary. Arrays are not types in Fortran. Being an array is a property of an object or function, not of a type. Unlike C, one cannot have an array of arrays or an array of pointers, -although can can have an array of a derived type that has arrays or +although one can have an array of a derived type that has arrays or pointers as components. Arrays are multidimensional, and the number of dimensions is called the _rank_ of the array. From 7a41761407c485d18b7d48232b308556b3b43934 Mon Sep 17 00:00:00 2001 From: Wanyi Date: Tue, 15 Apr 2025 13:46:15 -0400 Subject: [PATCH 02/63] [lldb] Make SBProcess thread related actions listen to StopLocker (#134339) # Summary This PR updates `SBProcess::GetNumThreads()` and `SBProcess::GetThreadAtIndex()` to listen to the stop locker. `SBProcess::GetNumThreads()` will return 0 if the process is running. ## Problem Description Recently upon debugging a program with thousands of threads in VS Code, lldb-dap would hang at a `threads` request sent right after receiving the `configurationDone` response. Soon after it will end the debug session with the following error ``` Process exited with status = -1 (0xffffffff) lost connection ``` This is because LLDB is still in the middle of resuming all the threads. And requesting threads will end up interrupt the process on Linux. From the gdb-remote log it ended up getting `lldb::StateType::eStateInvalid` and just exit with status -1. I don't think it's reasonable to allow getting threads from a running process. There are a few approaches to fix this: 1) Send the stopped event to IDE after `configurationDone`. This aligns with the CLI behavior. 2) However, the above approach will break the existing user facing behavior. The alternative will be reject the `threads` request if the process is not stopped. 3) Improve the run lock. This is a synchronize issue where process was in the middle of resuming while lldb-dap attempts to interrupt it. **This PR implements the option 3** ## HOWEVER This fixed the "lost connection" issue below but new issue has surfaced. From testing, and also from checking the [VSCode source code](https://github.com/microsoft/vscode/blob/174af221c9ea2ccdb64abe4aab8e1a805e77beae/src/vs/workbench/contrib/debug/browser/debugSession.ts#L791), it expects having threadID to perform `pause`. So after attaching, without any threads reported to the client, the user will not be able to pause the attached process. `setBreakpoint` will still work and once we make a stop at the bp (or any stop that will report threads, client can perform pause again. ## NEXT 1) Made an attempt to return initial thread list so that VSCode can pause (second commit in the PR) 2) Investigate why threads will trigger unwinding the second frame of a thread, which leads to sending the interrupt 3) Decided if we want to support `stopOnEntry` for attaching, given i. This is not an official specification ii. If enable stopOnEntry, we need to fix attaching on Linux, to send only one stopped event. Currently, all threads upon attaching will have stop reason `SIGSTOP` and lldb-dap will send `stopped` event for each one of them. Every `stopped` will trigger the client request for threads. iii. Alternatively, we can support auto continue correspond to `(lldb) process attach --continue`. This require the ii above. ### Additionally lldb-dap will not send a `continued` event after `configurationDone` because it checks `dap.focus_tid == LLDB_INVALID_THREAD_ID` (so that we don't send it for `launch` request). Notice `dap.focus_tid` will only get assigned when handling stop or stepping. According to DAP > Please note: a debug adapter is not expected to send this event in response to a request that implies that execution continues, e.g. launch or continue. It is only necessary to send a continued event if there was no previous request that implied this. So I guess we are not violating DAP if we don't send `continued` event. But I'd like to get some sense about this. ## Test Plan Used following program for testing: https://gist.github.com/kusmour/1729d2e07b7b1063897db77de194e47d **NOTE: Utilize stdin to get pid and attach AFTER hitting enter. Attach should happen when all the threads start running.** DAP messages before the change image DAP message after the change - report zero threads after attaching image --------- Co-authored-by: Jonas Devlieghere --- .../test/tools/lldb-dap/dap_server.py | 4 ++++ lldb/source/API/SBProcess.cpp | 20 ++++++++++--------- .../tools/lldb-dap/attach/TestDAP_attach.py | 2 +- lldb/tools/lldb-dap/DAP.h | 3 +++ .../ConfigurationDoneRequestHandler.cpp | 10 +++++++++- .../Handler/ThreadsRequestHandler.cpp | 17 +++++++++++----- lldb/tools/lldb-dap/JSONUtils.cpp | 13 ++++++++++++ lldb/tools/lldb-dap/JSONUtils.h | 2 ++ 8 files changed, 55 insertions(+), 16 deletions(-) diff --git a/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/dap_server.py b/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/dap_server.py index 45403e9df8525..61d7fa94479b8 100644 --- a/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/dap_server.py +++ b/lldb/packages/Python/lldbsuite/test/tools/lldb-dap/dap_server.py @@ -649,6 +649,10 @@ def request_configurationDone(self): response = self.send_recv(command_dict) if response: self.configuration_done_sent = True + # Client requests the baseline of currently existing threads after + # a successful launch or attach. + # Kick off the threads request that follows + self.request_threads() return response def _process_stopped(self): diff --git a/lldb/source/API/SBProcess.cpp b/lldb/source/API/SBProcess.cpp index 23ea449b30cca..ba77b2beed5ea 100644 --- a/lldb/source/API/SBProcess.cpp +++ b/lldb/source/API/SBProcess.cpp @@ -193,10 +193,11 @@ uint32_t SBProcess::GetNumThreads() { if (process_sp) { Process::StopLocker stop_locker; - const bool can_update = stop_locker.TryLock(&process_sp->GetRunLock()); - std::lock_guard guard( - process_sp->GetTarget().GetAPIMutex()); - num_threads = process_sp->GetThreadList().GetSize(can_update); + if (stop_locker.TryLock(&process_sp->GetRunLock())) { + std::lock_guard guard( + process_sp->GetTarget().GetAPIMutex()); + num_threads = process_sp->GetThreadList().GetSize(); + } } return num_threads; @@ -393,11 +394,12 @@ SBThread SBProcess::GetThreadAtIndex(size_t index) { ProcessSP process_sp(GetSP()); if (process_sp) { Process::StopLocker stop_locker; - const bool can_update = stop_locker.TryLock(&process_sp->GetRunLock()); - std::lock_guard guard( - process_sp->GetTarget().GetAPIMutex()); - thread_sp = process_sp->GetThreadList().GetThreadAtIndex(index, can_update); - sb_thread.SetThread(thread_sp); + if (stop_locker.TryLock(&process_sp->GetRunLock())) { + std::lock_guard guard( + process_sp->GetTarget().GetAPIMutex()); + thread_sp = process_sp->GetThreadList().GetThreadAtIndex(index, false); + sb_thread.SetThread(thread_sp); + } } return sb_thread; diff --git a/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py b/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py index 9df44cc454d5d..b9fbf2c8d14f9 100644 --- a/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py +++ b/lldb/test/API/tools/lldb-dap/attach/TestDAP_attach.py @@ -1,5 +1,5 @@ """ -Test lldb-dap setBreakpoints request +Test lldb-dap attach request """ diff --git a/lldb/tools/lldb-dap/DAP.h b/lldb/tools/lldb-dap/DAP.h index 8d32a18fb711e..b79a0d9d0f25c 100644 --- a/lldb/tools/lldb-dap/DAP.h +++ b/lldb/tools/lldb-dap/DAP.h @@ -211,6 +211,9 @@ struct DAP { /// The set of features supported by the connected client. llvm::DenseSet clientFeatures; + /// The initial thread list upon attaching. + std::optional initial_thread_list; + /// Creates a new DAP sessions. /// /// \param[in] log diff --git a/lldb/tools/lldb-dap/Handler/ConfigurationDoneRequestHandler.cpp b/lldb/tools/lldb-dap/Handler/ConfigurationDoneRequestHandler.cpp index cd120e1fdfaba..f39bbdefdbb95 100644 --- a/lldb/tools/lldb-dap/Handler/ConfigurationDoneRequestHandler.cpp +++ b/lldb/tools/lldb-dap/Handler/ConfigurationDoneRequestHandler.cpp @@ -44,6 +44,7 @@ namespace lldb_dap { // just an acknowledgement, so no body field is required." // }] // }, + void ConfigurationDoneRequestHandler::operator()( const llvm::json::Object &request) const { llvm::json::Object response; @@ -52,8 +53,15 @@ void ConfigurationDoneRequestHandler::operator()( dap.configuration_done_sent = true; if (dap.stop_at_entry) SendThreadStoppedEvent(dap); - else + else { + // Client requests the baseline of currently existing threads after + // a successful launch or attach by sending a 'threads' request + // right after receiving the configurationDone response. + // Obtain the list of threads before we resume the process + dap.initial_thread_list = + GetThreads(dap.target.GetProcess(), dap.thread_format); dap.target.GetProcess().Continue(); + } } } // namespace lldb_dap diff --git a/lldb/tools/lldb-dap/Handler/ThreadsRequestHandler.cpp b/lldb/tools/lldb-dap/Handler/ThreadsRequestHandler.cpp index 2b857f7f6a02b..16d797c2ab327 100644 --- a/lldb/tools/lldb-dap/Handler/ThreadsRequestHandler.cpp +++ b/lldb/tools/lldb-dap/Handler/ThreadsRequestHandler.cpp @@ -50,16 +50,23 @@ namespace lldb_dap { // } void ThreadsRequestHandler::operator()( const llvm::json::Object &request) const { - lldb::SBProcess process = dap.target.GetProcess(); llvm::json::Object response; FillResponse(request, response); - const uint32_t num_threads = process.GetNumThreads(); llvm::json::Array threads; - for (uint32_t thread_idx = 0; thread_idx < num_threads; ++thread_idx) { - lldb::SBThread thread = process.GetThreadAtIndex(thread_idx); - threads.emplace_back(CreateThread(thread, dap.thread_format)); + // Client requests the baseline of currently existing threads after + // a successful launch or attach by sending a 'threads' request + // right after receiving the configurationDone response. + // If no thread has reported to the client, it prevents something + // like the pause request from working in the running state. + // Return the cache of initial threads as the process might have resumed + if (dap.initial_thread_list) { + threads = dap.initial_thread_list.value(); + dap.initial_thread_list.reset(); + } else { + threads = GetThreads(dap.target.GetProcess(), dap.thread_format); } + if (threads.size() == 0) { response["success"] = llvm::json::Value(false); } diff --git a/lldb/tools/lldb-dap/JSONUtils.cpp b/lldb/tools/lldb-dap/JSONUtils.cpp index 7660403666150..33f10c93d2ada 100644 --- a/lldb/tools/lldb-dap/JSONUtils.cpp +++ b/lldb/tools/lldb-dap/JSONUtils.cpp @@ -870,6 +870,19 @@ llvm::json::Value CreateThread(lldb::SBThread &thread, lldb::SBFormat &format) { return llvm::json::Value(std::move(object)); } +llvm::json::Array GetThreads(lldb::SBProcess process, lldb::SBFormat &format) { + lldb::SBMutex lock = process.GetTarget().GetAPIMutex(); + std::lock_guard guard(lock); + + llvm::json::Array threads; + const uint32_t num_threads = process.GetNumThreads(); + for (uint32_t thread_idx = 0; thread_idx < num_threads; ++thread_idx) { + lldb::SBThread thread = process.GetThreadAtIndex(thread_idx); + threads.emplace_back(CreateThread(thread, format)); + } + return threads; +} + // "StoppedEvent": { // "allOf": [ { "$ref": "#/definitions/Event" }, { // "type": "object", diff --git a/lldb/tools/lldb-dap/JSONUtils.h b/lldb/tools/lldb-dap/JSONUtils.h index da91797290ff0..b8c53353bf42d 100644 --- a/lldb/tools/lldb-dap/JSONUtils.h +++ b/lldb/tools/lldb-dap/JSONUtils.h @@ -414,6 +414,8 @@ llvm::json::Value CreateExtendedStackFrameLabel(lldb::SBThread &thread, /// definition outlined by Microsoft. llvm::json::Value CreateThread(lldb::SBThread &thread, lldb::SBFormat &format); +llvm::json::Array GetThreads(lldb::SBProcess process, lldb::SBFormat &format); + /// Create a "StoppedEvent" object for a LLDB thread object. /// /// This function will fill in the following keys in the returned From 9b13d345303d819bb83de7ebbeb826d704add0bc Mon Sep 17 00:00:00 2001 From: Alexey Samsonov Date: Tue, 15 Apr 2025 10:49:52 -0700 Subject: [PATCH 03/63] [libc][bazel] Remove a no-op libc_internal_target macro. (#135818) This macro is a no-op after 90c001ac9e1d92a1a95d191d1640ab5337a937e5: libc_function macro now produce a "regular" cc_library target, without modifying its name, and this target is intended to only be used in tests. Thus, libc_internal_target macro is no longer needed, and we can safely treat libc_function rules and libc_support_library rules identically for test purposes. `libc_function_deps` attribute of a `libc_test` macro can also be cleaned up, but I plan to do this in a subsequent change. --- .../llvm-project-overlay/libc/libc_build_rules.bzl | 9 +-------- .../libc/test/libc_test_rules.bzl | 12 +++++------- 2 files changed, 6 insertions(+), 15 deletions(-) diff --git a/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl b/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl index 86dfb53a86014..60add23a46c48 100644 --- a/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl +++ b/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl @@ -10,10 +10,6 @@ load(":libc_configure_options.bzl", "LIBC_CONFIGURE_OPTIONS") load(":libc_namespace.bzl", "LIBC_NAMESPACE") load(":platforms.bzl", "PLATFORM_CPU_X86_64") -# TODO: Remove this helper function once all donwstream users are migrated. -def libc_internal_target(name): - return name - def libc_common_copts(): root_label = Label(":libc") libc_include_path = paths.join(root_label.workspace_root, root_label.package) @@ -84,10 +80,7 @@ def libc_function(name, **kwargs): # Builds "internal" library with a function, exposed as a C++ function in # the "LIBC_NAMESPACE" namespace. This allows us to test the function in the # presence of another libc. - _libc_library( - name = libc_internal_target(name), - **kwargs - ) + _libc_library(name = name, **kwargs) LibcLibraryInfo = provider( "All source files and textual headers for building a particular library.", diff --git a/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl b/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl index 8c20a9172989c..7e798429ef19b 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl +++ b/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl @@ -12,26 +12,24 @@ They come in two flavors: When performing tests we make sure to always use the internal version. """ -load("//libc:libc_build_rules.bzl", "libc_common_copts", "libc_internal_target") +load("//libc:libc_build_rules.bzl", "libc_common_copts") load("//libc:libc_configure_options.bzl", "LIBC_CONFIGURE_OPTIONS") -def libc_test(name, srcs, libc_function_deps = [], copts = [], deps = [], local_defines = [], **kwargs): +def libc_test(name, libc_function_deps = [], copts = [], deps = [], local_defines = [], **kwargs): """Add target for a libc test. Args: name: Test target name - srcs: List of sources for the test. libc_function_deps: List of libc_function targets used by this test. copts: The list of options to add to the C++ compilation command. deps: The list of other libraries to be linked in to the test target. local_defines: The list of target local_defines if any. - **kwargs: Attributes relevant for a libc_test. For example, name, srcs. + **kwargs: Attributes relevant for a cc_test. """ native.cc_test( name = name, - srcs = srcs, local_defines = local_defines + LIBC_CONFIGURE_OPTIONS, - deps = [libc_internal_target(d) for d in libc_function_deps] + [ + deps = [ "//libc/test/UnitTest:LibcUnitTest", "//libc:__support_macros_config", "//libc:errno", @@ -39,7 +37,7 @@ def libc_test(name, srcs, libc_function_deps = [], copts = [], deps = [], local_ "//libc:func_free", "//libc:func_malloc", "//libc:func_realloc", - ] + deps, + ] + libc_function_deps + deps, copts = copts + libc_common_copts(), linkstatic = 1, **kwargs From 3b9103044361094a8fde16a877f2e8cb0f96ce24 Mon Sep 17 00:00:00 2001 From: Andrew Rogers Date: Tue, 15 Apr 2025 10:51:02 -0700 Subject: [PATCH 04/63] [llvm] add documentation for public interface annotations (LLVM_ABI, etc) ## Purpose Add documentation for the existing family of `LLVM_ABI` annotation macros defined in llvm/Support/Compiler.h. These annotations are used to describe LLVM's public interface. ## Background This documentation is in support of the annotation effort described [here](https://discourse.llvm.org/t/psa-annotating-llvm-public-interface/85307/). ## Validation Manually inspected rendered ReST document on GitHub. Co-authored-by: Saleem Abdulrasool --- llvm/docs/InterfaceExportAnnotations.rst | 376 +++++++++++++++++++++++ llvm/docs/Reference.rst | 1 + 2 files changed, 377 insertions(+) create mode 100644 llvm/docs/InterfaceExportAnnotations.rst diff --git a/llvm/docs/InterfaceExportAnnotations.rst b/llvm/docs/InterfaceExportAnnotations.rst new file mode 100644 index 0000000000000..eecf6ffe6eaca --- /dev/null +++ b/llvm/docs/InterfaceExportAnnotations.rst @@ -0,0 +1,376 @@ +LLVM Interface Export Annotations +================================= +Symbols that are part of LLVM's public interface must be explicitly annotated +to support shared library builds with hidden default symbol visibility. This +document provides background and guidelines for annotating the codebase. + +LLVM Shared Library +------------------- +LLVM builds as a static library by default, but it can also be built as a shared +library with the following configuration: + +:: + + LLVM_BUILD_LLVM_DYLIB=On + LLVM_LINK_LLVM_DYLIB=On + +There are three shared library executable formats we're interested in: PE +Dynamic Link Library (.dll) on Windows, Mach-O Shared Object (.dylib) on Apple +systems, and ELF Shared Object (.so) on Linux, BSD and other Unix-like systems. + +ELF and Mach-O Shared Object files can be built with no additional setup or +configuration. This is because all global symbols in the library are exported by +default -- the same as when building a static library. However, when building a +DLL for Windows, the situation is more complex: + +- Symbols are not exported from a DLL by default. Symbols must be annotated with + ``__declspec(dllexport)`` when building the library to be externally visible. + +- Symbols imported from a Windows DLL should generally be annotated with + ``__declspec(dllimport)`` when compiling clients. + +- A single Windows DLL can export a maximum of 65,535 symbols. + +Because of the requirements for Windows DLLs, additional work must be done to +ensure the proper set of public symbols is exported and visible to clients. + +Annotation Macros +----------------- +The distinct DLL import and export annotations required for Windows DLLs +typically lead developers to define a preprocessor macro for annotating exported +symbols in header public files. The custom macro resolves to the _export_ +annotation when building the library and the _import_ annotation when building +the client. + +We have defined the `LLVM_ABI` macro in `llvm/Support/Compiler.h +`__ +for this purpose: + +.. code:: cpp + + #if defined(LLVM_EXPORTS) + #define LLVM_ABI __declspec(dllexport) + #else + #define LLVM_ABI __declspec(dllimport) + #endif + +Windows DLL symbol visibility requirements are approximated on ELF and Mach-O +shared library builds by setting default symbol visibility to hidden +(``-fvisibility-default=hidden``) when building with the following +configuration: + +:: + + LLVM_BUILD_LLVM_DYLIB_VIS=On + +For an ELF or Mach-O platform with this setting, the ``LLVM_ABI`` macro is +defined to override the default hidden symbol visibility: + +.. code:: cpp + + #define LLVM_ABI __attribute__((visibility("default"))) + +In addition to ``LLVM_ABI``, there are a few other macros for use in less +common cases described below. + +Export macros are used to annotate symbols only within their intended shared +library. This is necessary because of the way Windows handles import/export +annotations. + +For example, ``LLVM_ABI`` resolves to ``__declspec(dllexport)`` only when +building source that is part of the LLVM shared library (e.g. source under +``llvm-project/llvm``). If ``LLVM_ABI`` were incorrectly used to annotate a +symbol from a different LLVM project (such as Clang) it would always resolve to +``__declspec(dllimport)`` and the symbol would not be properly exported. + +Annotating Symbols +------------------ +Functions +~~~~~~~~~ +Exported function declarations in header files must be annotated with +``LLVM_ABI``. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + LLVM_ABI void exported_function(int a, int b); + +Global Variables +~~~~~~~~~~~~~~~~ +Exported global variables must be annotated with ``LLVM_ABI`` at their +``extern`` declarations. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + LLVM_ABI extern int exported_global_variable; + +Classes, Structs, and Unions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Classes, structs, and unions can be annotated with ``LLVM_ABI`` at their +declaration, but this option is generally discouraged because it will +export every class member, vtable, and type information. Instead, ``LLVM_ABI`` +should be applied to individual class members that require export. + +In the most common case, public and protected methods without a body in the +class declaration must be annotated with ``LLVM_ABI``. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + class ExampleClass { + public: + // Public methods defined externally must be annotatated. + LLVM_ABI int sourceDefinedPublicMethod(int a, int b); + + // Methods defined in the class definition do not need annotation. + int headerDefinedPublicMethod(int a, int b) { + return a + b; + } + + // Constructors and destructors must be annotated if defined externally. + ExampleClass() {} + LLVM_ABI ~ExampleClass(); + + // Public static methods defined externally must be annotatated. + LLVM_ABI static int sourceDefinedPublicStaticMethod(int a, int b); + }; + +Additionally, public and protected static fields that are not initialized at +declaration must be annotated with ``LLVM_ABI``. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + class ExampleClass { + public: + // Public static fields defined externally must be annotated. + LLVM_ABI static int mutableStaticField; + LLVM_ABI static const int constStaticField; + + // Static members initialized at declaration do not need to be annotated. + static const int initializedConstStaticField = 0; + static constexpr int initializedConstexprStaticField = 0; + }; + +Private methods may also require ``LLVM_ABI`` annotation in certain cases. This +situation occurs when a method defined in a header calls the private method. The +private method call may be from within the class, a parent class, or a friend +class. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + class ExampleClass { + private: + // Private methods must be annotated if referenced by a public method defined a + // header file. + LLVM_ABI int privateMethod(int a, int b); + + public: + // Inlineable method defined in the class definition calls a private method + // defined externally. If the private method is not annotated for export, this + // method will fail to link. + int publicMethod(int a, int b) { + return privateMethod(a, b); + } + }; + +There are less common cases where you may also need to annotate an inline +function even though it is fully defined in a header. Annotating an inline +function for export does not prevent it being inlined into client code. However, +it does ensure there is a single, stable address for the function exported from +the shared library. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + // Annotate the function so it is exported from the library at a fixed + // address. + LLVM_ABI inline int inlineFunction(int a, int b) { + return a + b; + } + +Similarly, if a stable pointer-to-member function address is required for a +method in a C++ class, it may be annotated for export. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + class ExampleClass { + public: + // Annotate the method so it is exported from the library at a fixed + // address. + LLVM_ABI inline int inlineMethod(int a, int b) { + return a + b; + } + }; + +.. note:: + + When an inline function is annotated for export, the header containing the + function definition **must** be included by at least one of the library's + source files or the function will never be compiled with the export + annotation. + +Friend Functions +~~~~~~~~~~~~~~~~ +Friend functions declared in a class, struct or union must be annotated with +``LLVM_ABI`` if the corresponding function declaration is also annotated. This +requirement applies even when the class itself is annotated with ``LLVM_ABI``. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + // An exported function that has friend access to ExampleClass internals. + LLVM_ABI int friend_function(ExampleClass &obj); + + class ExampleClass { + // Friend declaration of a function must be annotated the same as the actual + // function declaration. + LLVM_ABI friend int friend_function(ExampleClass &obj); + }; + +.. note:: + + Annotating the friend declaration avoids an “inconsistent dll linkage” + compiler error when building for Windows. This annotation is harmless but not + required when building ELF or Mach-O shared libraries. + +Virtual Table and Type Info +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Classes and structs with exported virtual methods, or child classes that export +overridden virtual methods, must also export their vtable for ELF and Mach-O +builds. This can be achieved by annotating the class rather than individual +class members. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + class ParentClass { + public: + virtual int virtualMethod(int a, int b); + virtual int anotherVirtualMethod(int a, int b); + virtual ~ParentClass(); + }; + + // Annotating the class exports vtable and type information as well as all + // class members. + class LLVM_ABI ChildClass : public ParentClass { + public: + // Inline method override does not require the class be annotated. + int virtualMethod(int a, int b) override { + return ParentClass::virtualMethod(a, b); + } + + // Overriding a virtual method from the parent requires the class be + // annotated. The parent class may require annotation as well. + int pureVirtualMethod(int a, int b) override; + ~ChildClass(); + }; + +If annotating a type with ``LLVM_ABI`` causes compilation issues such as those +described +`here `__, +the class may require modification. Often, explicitly deleting the copy +constructor and copy assignment operator will resolve the issue. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + #include + + class LLVM_ABI ExportedClass { + public: + // Explicitly delete the copy constructor and assignment operator. + ExportedClass(ExportedClass const&) = delete; + ExportedClass& operator=(ExportedClass const&) = delete; + }; + +Templates +~~~~~~~~~ +Most template classes are entirely header-defined and do not need to be exported +because they will be instantiated and compiled into the client as needed. Such +template classes require no export annotations. However, there are some less +common cases where annotations are required for templates. + +Specialized Template Functions +++++++++++++++++++++++++++++++ +As with any other exported function, an exported specialization of a template +function not defined in a header file must have its declaration annotated with +``LLVM_ABI``. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + template T templateMethod(T a, T b) { + return a + b; + } + + // The explicitly specialized definition of templateMethod for int is located in + // a source file. This declaration must be annotated with LLVM_ABI to export it. + template <> LLVM_ABI int templateMethod(int a, int b); + +Similarly, an exported specialization of a method in a template class must have +its declaration annotated with ``LLVM_ABI``. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + template class TemplateClass { + public: + int method(int a, int b) { + return a + b; + } + }; + + // The explicitly specialized definition of method for int is defined in a + // source file. The declaration must be annotated with LLVM_ABI to export it. + template <> LLVM_ABI int TemplateStruct::method(int a, int b); + +Explicitly Instantiated Template Classes +++++++++++++++++++++++++++++++++++++++++ +Explicitly instantiated template classes must be annotated with +template-specific annotations at both declaration and definition. + +An extern template instantiation in a header file must be annotated with +``LLVM_TEMPLATE_ABI``. This will typically be located in a header file. + +.. code:: cpp + + #include "llvm/Support/Compiler.h" + + template class TemplateClass { + public: + TemplateClass(T val) : val_(val) {} + + T get() const { return val_; } + + private: + const T val_; + }; + + // Explicitly instantiate and export TempalateClass for int type. + extern template class LLVM_TEMPLATE_ABI TemplateClass; + +The corresponding definition of the template instantiation must be annotated +with ``LLVM_EXPORT_TEMPLATE``. This will typically be located in a source file. + +.. code:: cpp + + #include "TemplateClass.h" + + // Explicitly instantiate and export TempalateClass for int type. + template class LLVM_EXPORT_TEMPLATE TemplateClass; diff --git a/llvm/docs/Reference.rst b/llvm/docs/Reference.rst index 470b6bd024b89..e1f46b00f2b30 100644 --- a/llvm/docs/Reference.rst +++ b/llvm/docs/Reference.rst @@ -31,6 +31,7 @@ LLVM and API reference documentation. HowToSetUpLLVMStyleRTTI HowToUseAttributes InAlloca + InterfaceExportAnnotations LangRef LibFuzzer MarkedUpDisassembly From 30d13e359190f7a0e2122292ec4a4fc1a6c71acc Mon Sep 17 00:00:00 2001 From: Amr Hesham Date: Tue, 15 Apr 2025 19:52:26 +0200 Subject: [PATCH 05/63] [CIR] Upstream ArraySubscriptExpr from function parameter with pointer base (#135493) This change adds an ArraySubscriptExpr from the function parameter with base type as Pointer Issue https://github.com/llvm/llvm-project/issues/130197 --- clang/lib/CIR/CodeGen/CIRGenExpr.cpp | 43 ++++++++-- clang/test/CIR/CodeGen/array.cpp | 122 ++++++++++++++++++++++++--- 2 files changed, 148 insertions(+), 17 deletions(-) diff --git a/clang/lib/CIR/CodeGen/CIRGenExpr.cpp b/clang/lib/CIR/CodeGen/CIRGenExpr.cpp index f0732a8ea60af..cffe5c5cd1ec3 100644 --- a/clang/lib/CIR/CodeGen/CIRGenExpr.cpp +++ b/clang/lib/CIR/CodeGen/CIRGenExpr.cpp @@ -552,7 +552,19 @@ CIRGenFunction::emitArraySubscriptExpr(const clang::ArraySubscriptExpr *e) { // in lexical order (this complexity is, sadly, required by C++17). assert((e->getIdx() == e->getLHS() || e->getIdx() == e->getRHS()) && "index was neither LHS nor RHS"); - const mlir::Value idx = emitScalarExpr(e->getIdx()); + + auto emitIdxAfterBase = [&]() -> mlir::Value { + const mlir::Value idx = emitScalarExpr(e->getIdx()); + + // Extend or truncate the index type to 32 or 64-bits. + auto ptrTy = mlir::dyn_cast(idx.getType()); + if (ptrTy && mlir::isa(ptrTy.getPointee())) + cgm.errorNYI(e->getSourceRange(), + "emitArraySubscriptExpr: index type cast"); + return idx; + }; + + const mlir::Value idx = emitIdxAfterBase(); if (const Expr *array = getSimpleArrayDecayOperand(e->getBase())) { LValue arrayLV; if (const auto *ase = dyn_cast(array)) @@ -566,13 +578,34 @@ CIRGenFunction::emitArraySubscriptExpr(const clang::ArraySubscriptExpr *e) { arrayLV.getAddress(), e->getType(), idx, cgm.getLoc(e->getExprLoc()), /*shouldDecay=*/true); - return LValue::makeAddr(addr, e->getType(), LValueBaseInfo()); + const LValue lv = LValue::makeAddr(addr, e->getType(), LValueBaseInfo()); + + if (getLangOpts().ObjC && getLangOpts().getGC() != LangOptions::NonGC) { + cgm.errorNYI(e->getSourceRange(), "emitArraySubscriptExpr: ObjC with GC"); + } + + return lv; } // The base must be a pointer; emit it with an estimate of its alignment. - cgm.errorNYI(e->getSourceRange(), - "emitArraySubscriptExpr: The base must be a pointer"); - return {}; + assert(e->getBase()->getType()->isPointerType() && + "The base must be a pointer"); + + LValueBaseInfo eltBaseInfo; + const Address ptrAddr = emitPointerWithAlignment(e->getBase(), &eltBaseInfo); + // Propagate the alignment from the array itself to the result. + const Address addxr = emitArraySubscriptPtr( + *this, cgm.getLoc(e->getBeginLoc()), cgm.getLoc(e->getEndLoc()), ptrAddr, + e->getType(), idx, cgm.getLoc(e->getExprLoc()), + /*shouldDecay=*/false); + + const LValue lv = LValue::makeAddr(addxr, e->getType(), eltBaseInfo); + + if (getLangOpts().ObjC && getLangOpts().getGC() != LangOptions::NonGC) { + cgm.errorNYI(e->getSourceRange(), "emitArraySubscriptExpr: ObjC with GC"); + } + + return lv; } LValue CIRGenFunction::emitBinaryOperatorLValue(const BinaryOperator *e) { diff --git a/clang/test/CIR/CodeGen/array.cpp b/clang/test/CIR/CodeGen/array.cpp index 5cda061cdbf12..08f6d730f161a 100644 --- a/clang/test/CIR/CodeGen/array.cpp +++ b/clang/test/CIR/CodeGen/array.cpp @@ -350,20 +350,118 @@ void func7() { // OGCG: %[[ARR:.*]] = alloca [1 x ptr], align 8 // OGCG: call void @llvm.memset.p0.i64(ptr align 8 %[[ARR]], i8 0, i64 8, i1 false) -void func8(int p[10]) {} -// CIR: cir.func @func8(%arg0: !cir.ptr -// CIR: cir.alloca !cir.ptr, !cir.ptr>, ["p", init] +void func8(int arr[10]) { + int e = arr[0]; + int e2 = arr[1]; +} -// LLVM: define void @func8(ptr {{%.*}}) -// LLVM-NEXT: alloca ptr, i64 1, align 8 +// CIR: cir.func @func8(%[[ARG:.*]]: !cir.ptr +// CIR: %[[ARR:.*]] = cir.alloca !cir.ptr, !cir.ptr>, ["arr", init] +// CIR: %[[INIT:.*]] = cir.alloca !s32i, !cir.ptr, ["e", init] +// CIR: %[[INIT_2:.*]] = cir.alloca !s32i, !cir.ptr, ["e2", init] +// CIR: cir.store %[[ARG]], %[[ARR]] : !cir.ptr, !cir.ptr> +// CIR: %[[IDX:.*]] = cir.const #cir.int<0> : !s32i +// CIR: %[[TMP_1:.*]] = cir.load %[[ARR]] : !cir.ptr>, !cir.ptr +// CIR: %[[ELE_0:.*]] = cir.ptr_stride(%[[TMP_1]] : !cir.ptr, %[[IDX]] : !s32i), !cir.ptr +// CIR: %[[TMP_2:.*]] = cir.load %[[ELE_0]] : !cir.ptr, !s32i +// CIR: cir.store %[[TMP_2]], %[[INIT]] : !s32i, !cir.ptr +// CIR: %[[IDX_1:.*]] = cir.const #cir.int<1> : !s32i +// CIR: %[[TMP_3:.*]] = cir.load %[[ARR]] : !cir.ptr>, !cir.ptr +// CIR: %[[ELE_1:.*]] = cir.ptr_stride(%[[TMP_3]] : !cir.ptr, %[[IDX_1]] : !s32i), !cir.ptr +// CIR: %[[TMP_4:.*]] = cir.load %[[ELE_1]] : !cir.ptr, !s32i +// CIR: cir.store %[[TMP_4]], %[[INIT_2]] : !s32i, !cir.ptr + +// LLVM: define void @func8(ptr %[[ARG:.*]]) +// LLVM: %[[ARR:.*]] = alloca ptr, i64 1, align 8 +// LLVM: %[[INIT:.*]] = alloca i32, i64 1, align 4 +// LLVM: %[[INIT_2:.*]] = alloca i32, i64 1, align 4 +// LLVM: store ptr %[[ARG]], ptr %[[ARR]], align 8 +// LLVM: %[[TMP_1:.*]] = load ptr, ptr %[[ARR]], align 8 +// LLVM: %[[ELE_0:.*]] = getelementptr i32, ptr %[[TMP_1]], i64 0 +// LLVM: %[[TMP_2:.*]] = load i32, ptr %[[ELE_0]], align 4 +// LLVM: store i32 %[[TMP_2]], ptr %[[INIT]], align 4 +// LLVM: %[[TMP_3:.*]] = load ptr, ptr %[[ARR]], align 8 +// LLVM: %[[ELE_1:.*]] = getelementptr i32, ptr %[[TMP_3]], i64 1 +// LLVM: %[[TMP_4:.*]] = load i32, ptr %[[ELE_1]], align 4 +// LLVM: store i32 %[[TMP_4]], ptr %[[INIT_2]], align 4 + +// OGCG: %[[ARR:.*]] = alloca ptr, align 8 +// OGCG: %[[INIT:.*]] = alloca i32, align 4 +// OGCG: %[[INIT_2:.*]] = alloca i32, align 4 +// OGCG: store ptr {{%.*}}, ptr %[[ARR]], align 8 +// OGCG: %[[TMP_1:.*]] = load ptr, ptr %[[ARR]], align 8 +// OGCG: %[[ELE_0:.*]] = getelementptr inbounds i32, ptr %[[TMP_1]], i64 0 +// OGCG: %[[TMP_2:.*]] = load i32, ptr %[[ELE_0]], align 4 +// OGCG: store i32 %[[TMP_2]], ptr %[[INIT]], align 4 +// OGCG: %[[TMP_3:.*]] = load ptr, ptr %[[ARR]], align 8 +// OGCG: %[[ELE_1:.*]] = getelementptr inbounds i32, ptr %[[TMP_3]], i64 1 +// OGCG: %[[TMP_2:.*]] = load i32, ptr %[[ELE_1]], align 4 +// OGCG: store i32 %[[TMP_2]], ptr %[[INIT_2]], align 4 -// OGCG: alloca ptr, align 8 +void func9(int arr[10][5]) { + int e = arr[1][2]; +} -void func9(int pp[10][5]) {} -// CIR: cir.func @func9(%arg0: !cir.ptr> -// CIR: cir.alloca !cir.ptr>, !cir.ptr>> +// CIR: cir.func @func9(%[[ARG:.*]]: !cir.ptr> +// CIR: %[[ARR:.*]] = cir.alloca !cir.ptr>, !cir.ptr>>, ["arr", init] +// CIR: %[[INIT:.*]] = cir.alloca !s32i, !cir.ptr, ["e", init] +// CIR: cir.store %[[ARG]], %[[ARR]] : !cir.ptr>, !cir.ptr>> +// CIR: %[[IDX:.*]] = cir.const #cir.int<2> : !s32i +// CIR: %[[IDX_1:.*]] = cir.const #cir.int<1> : !s32i +// CIR: %[[TMP_1:.*]] = cir.load %[[ARR]] : !cir.ptr>>, !cir.ptr> +// CIR: %[[ARR_1:.*]] = cir.ptr_stride(%[[TMP_1]] : !cir.ptr>, %[[IDX_1]] : !s32i), !cir.ptr> +// CIR: %[[ARR_1_PTR:.*]] = cir.cast(array_to_ptrdecay, %[[ARR_1]] : !cir.ptr>), !cir.ptr +// CIR: %[[ARR_1_2:.*]] = cir.ptr_stride(%[[ARR_1_PTR]] : !cir.ptr, %[[IDX]] : !s32i), !cir.ptr +// CIR: %[[TMP_2:.*]] = cir.load %[[ARR_1_2]] : !cir.ptr, !s32i +// CIR: cir.store %[[TMP_2]], %[[INIT]] : !s32i, !cir.ptr + +// LLVM: define void @func9(ptr %[[ARG:.*]]) +// LLVM: %[[ARR:.*]] = alloca ptr, i64 1, align 8 +// LLVM: %[[INIT:.*]] = alloca i32, i64 1, align 4 +// LLVM: store ptr %[[ARG]], ptr %[[ARR]], align 8 +// LLVM: %[[TMP_1:.*]] = load ptr, ptr %[[ARR]], align 8 +// LLVM: %[[ARR_1:.*]] = getelementptr [5 x i32], ptr %[[TMP_1]], i64 1 +// LLVM: %[[ARR_1_PTR:.*]] = getelementptr i32, ptr %[[ARR_1]], i32 0 +// LLVM: %[[ARR_1_2:.*]] = getelementptr i32, ptr %[[ARR_1_PTR]], i64 2 +// LLVM: %[[TMP_2:.*]] = load i32, ptr %[[ARR_1_2]], align 4 +// LLVM: store i32 %[[TMP_2]], ptr %[[INIT]], align 4 + +// OGCG: %[[ARR:.*]] = alloca ptr, align 8 +// OGCG: %[[INIT:.*]] = alloca i32, align 4 +// OGCG: store ptr {{%.*}}, ptr %[[ARR]], align 8 +// OGCG: %[[TMP_1:.*]] = load ptr, ptr %[[ARR]], align 8 +// OGCG: %[[ARR_1:.*]] = getelementptr inbounds [5 x i32], ptr %[[TMP_1]], i64 1 +// OGCG: %[[ARR_1_2:.*]] = getelementptr inbounds [5 x i32], ptr %[[ARR_1]], i64 0, i64 2 +// OGCG: %[[TMP_2:.*]] = load i32, ptr %[[ARR_1_2]], align 4 +// OGCG: store i32 %[[TMP_2]], ptr %[[INIT]], align 4 + +void func10(int *a) { + int e = a[5]; +} -// LLVM: define void @func9(ptr {{%.*}}) -// LLVM-NEXT: alloca ptr, i64 1, align 8 +// CIR: cir.func @func10(%[[ARG:.*]]: !cir.ptr +// CIR: %[[ARR:.*]] = cir.alloca !cir.ptr, !cir.ptr>, ["a", init] +// CIR: %[[INIT:.*]] = cir.alloca !s32i, !cir.ptr, ["e", init] +// CIR: cir.store %[[ARG]], %[[ARR]] : !cir.ptr, !cir.ptr> +// CIR: %[[IDX:.*]] = cir.const #cir.int<5> : !s32i +// CIR: %[[TMP_1:.*]] = cir.load %[[ARR]] : !cir.ptr>, !cir.ptr +// CIR: %[[ELE:.*]] = cir.ptr_stride(%[[TMP_1]] : !cir.ptr, %[[IDX]] : !s32i), !cir.ptr +// CIR: %[[TMP_2:.*]] = cir.load %[[ELE]] : !cir.ptr, !s32i +// CIR: cir.store %[[TMP_2]], %[[INIT]] : !s32i, !cir.ptr + +// LLVM: define void @func10(ptr %[[ARG:.*]]) { +// LLVM: %[[ARR:.*]] = alloca ptr, i64 1, align 8 +// LLVM: %[[INIT:.*]] = alloca i32, i64 1, align 4 +// LLVM: store ptr %[[ARG]], ptr %[[ARR]], align 8 +// LLVM: %[[TMP_1:.*]] = load ptr, ptr %[[ARR]], align 8 +// LLVM: %[[ELE:.*]] = getelementptr i32, ptr %[[TMP_1]], i64 5 +// LLVM: %[[TMP_2:.*]] = load i32, ptr %[[ELE]], align 4 +// LLVM: store i32 %[[TMP_2]], ptr %[[INIT]], align 4 -// OGCG: alloca ptr, align 8 +// OGCG: %[[ARR:.*]] = alloca ptr, align 8 +// OGCG: %[[INIT:.*]] = alloca i32, align 4 +// OGCG: store ptr {{%.*}}, ptr %[[ARR]], align 8 +// OGCG: %[[TMP_1:.*]] = load ptr, ptr %[[ARR]], align 8 +// OGCG: %[[ELE:.*]] = getelementptr inbounds i32, ptr %[[TMP_1]], i64 5 +// OGCG: %[[TMP_2:.*]] = load i32, ptr %[[ELE]], align 4 +// OGCG: store i32 %[[TMP_2]], ptr %[[INIT]], align 4 From 3f58ff20fe540fbbc2e5bfea1606f8cdc00d4157 Mon Sep 17 00:00:00 2001 From: Peter Collingbourne Date: Tue, 15 Apr 2025 11:08:48 -0700 Subject: [PATCH 06/63] AArch64: Remove the PAUTH_BLEND pseudo-instruction. It can be represented using a regular MOVK instruction which also has the advantage of sometimes being selectable without a preceding MOV. Reviewers: ahmedbougacha, asl, atrosinenko Reviewed By: atrosinenko Pull Request: https://github.com/llvm/llvm-project/pull/134765 --- llvm/lib/Target/AArch64/AArch64InstrInfo.td | 5 +-- .../lib/Target/AArch64/AArch64PointerAuth.cpp | 37 ------------------- .../AArch64/ptrauth-pseudo-instructions.mir | 27 -------------- 3 files changed, 1 insertion(+), 68 deletions(-) delete mode 100644 llvm/test/CodeGen/AArch64/ptrauth-pseudo-instructions.mir diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.td b/llvm/lib/Target/AArch64/AArch64InstrInfo.td index b90792d60d102..99f2b79d31bb7 100644 --- a/llvm/lib/Target/AArch64/AArch64InstrInfo.td +++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.td @@ -1798,9 +1798,6 @@ def PAUTH_PROLOGUE : Pseudo<(outs), (ins), []>, Sched<[]> { def PAUTH_EPILOGUE : Pseudo<(outs), (ins), []>, Sched<[]>; } -def PAUTH_BLEND : Pseudo<(outs GPR64:$disc), - (ins GPR64:$addr_disc, i32imm:$int_disc), []>, Sched<[]>; - // These pointer authentication instructions require armv8.3a let Predicates = [HasPAuth] in { @@ -10130,7 +10127,7 @@ let Predicates = [HasMOPS, HasMTE], Defs = [NZCV], Size = 12, mayLoad = 0, maySt // v8.3 Pointer Authentication late patterns def : Pat<(int_ptrauth_blend GPR64:$Rd, imm64_0_65535:$imm), - (PAUTH_BLEND GPR64:$Rd, (trunc_imm imm64_0_65535:$imm))>; + (MOVKXi GPR64:$Rd, (trunc_imm imm64_0_65535:$imm), 48)>; def : Pat<(int_ptrauth_blend GPR64:$Rd, GPR64:$Rn), (BFMXri GPR64:$Rd, GPR64:$Rn, 16, 15)>; diff --git a/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp b/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp index c3bc70ad6f427..ba03f4e257b69 100644 --- a/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp +++ b/llvm/lib/Target/AArch64/AArch64PointerAuth.cpp @@ -42,13 +42,6 @@ class AArch64PointerAuth : public MachineFunctionPass { void authenticateLR(MachineFunction &MF, MachineBasicBlock::iterator MBBI) const; - /// Stores blend(AddrDisc, IntDisc) to the Result register. - void emitBlend(MachineBasicBlock::iterator MBBI, Register Result, - Register AddrDisc, unsigned IntDisc) const; - - /// Expands PAUTH_BLEND pseudo instruction. - void expandPAuthBlend(MachineBasicBlock::iterator MBBI) const; - bool checkAuthenticatedLR(MachineBasicBlock::iterator TI) const; }; @@ -249,32 +242,6 @@ unsigned llvm::AArch64PAuth::getCheckerSizeInBytes(AuthCheckMethod Method) { llvm_unreachable("Unknown AuthCheckMethod enum"); } -void AArch64PointerAuth::emitBlend(MachineBasicBlock::iterator MBBI, - Register Result, Register AddrDisc, - unsigned IntDisc) const { - MachineBasicBlock &MBB = *MBBI->getParent(); - DebugLoc DL = MBBI->getDebugLoc(); - - if (Result != AddrDisc) - BuildMI(MBB, MBBI, DL, TII->get(AArch64::ORRXrs), Result) - .addReg(AArch64::XZR) - .addReg(AddrDisc) - .addImm(0); - - BuildMI(MBB, MBBI, DL, TII->get(AArch64::MOVKXi), Result) - .addReg(Result) - .addImm(IntDisc) - .addImm(48); -} - -void AArch64PointerAuth::expandPAuthBlend( - MachineBasicBlock::iterator MBBI) const { - Register ResultReg = MBBI->getOperand(0).getReg(); - Register AddrDisc = MBBI->getOperand(1).getReg(); - unsigned IntDisc = MBBI->getOperand(2).getImm(); - emitBlend(MBBI, ResultReg, AddrDisc, IntDisc); -} - bool AArch64PointerAuth::runOnMachineFunction(MachineFunction &MF) { Subtarget = &MF.getSubtarget(); TII = Subtarget->getInstrInfo(); @@ -290,7 +257,6 @@ bool AArch64PointerAuth::runOnMachineFunction(MachineFunction &MF) { break; case AArch64::PAUTH_PROLOGUE: case AArch64::PAUTH_EPILOGUE: - case AArch64::PAUTH_BLEND: PAuthPseudoInstrs.push_back(MI.getIterator()); break; } @@ -305,9 +271,6 @@ bool AArch64PointerAuth::runOnMachineFunction(MachineFunction &MF) { case AArch64::PAUTH_EPILOGUE: authenticateLR(MF, It); break; - case AArch64::PAUTH_BLEND: - expandPAuthBlend(It); - break; default: llvm_unreachable("Unhandled opcode"); } diff --git a/llvm/test/CodeGen/AArch64/ptrauth-pseudo-instructions.mir b/llvm/test/CodeGen/AArch64/ptrauth-pseudo-instructions.mir deleted file mode 100644 index d7fe1953deb47..0000000000000 --- a/llvm/test/CodeGen/AArch64/ptrauth-pseudo-instructions.mir +++ /dev/null @@ -1,27 +0,0 @@ -# RUN: llc -mtriple=aarch64--- -run-pass=aarch64-ptrauth -verify-machineinstrs %s -o - | FileCheck %s - -# Test the corner cases that cannot be reliably tested using LLVM IR as input. - ---- | - define i64 @blend_untied(i64 %unused, i64 %ptr_arg) { - ret i64 0 - } -... ---- -# Check that the input register is copied to the output one, if not tied. - -name: blend_untied -tracksRegLiveness: true -body: | - bb.0: - liveins: $lr, $x0, $x1 - $x0 = PAUTH_BLEND $x1, 42 - RET undef $lr - -# CHECK: liveins: $lr, $x0, $x1 -# CHECK-NEXT: {{^ +$}} -# CHECK-NEXT: $x0 = ORRXrs $xzr, $x1, 0 -# CHECK-NEXT: $x0 = MOVKXi $x0, 42, 48 -# CHECK-NEXT: RET undef $lr - -... From a5aa0c46c3274eaf25dde4d792a1abd6191cccf9 Mon Sep 17 00:00:00 2001 From: Peter Collingbourne Date: Tue, 15 Apr 2025 11:12:05 -0700 Subject: [PATCH 07/63] Introduce -funique-source-file-names flag. The purpose of this flag is to allow the compiler to assume that each object file passed to the linker has been compiled using a unique source file name. This is useful for reducing link times when doing ThinLTO in combination with whole-program devirtualization or CFI, as it allows modules without exported symbols to be built with ThinLTO. Reviewers: vitalybuka, teresajohnson Reviewed By: teresajohnson Pull Request: https://github.com/llvm/llvm-project/pull/135728 --- clang/docs/ControlFlowIntegrity.rst | 5 +++ clang/docs/UsersManual.rst | 10 +++++ clang/include/clang/Basic/CodeGenOptions.def | 2 + clang/include/clang/Driver/Options.td | 7 ++++ clang/lib/CodeGen/CodeGenModule.cpp | 4 ++ clang/lib/Driver/ToolChains/Clang.cpp | 3 ++ clang/test/CodeGen/unique-source-file-names.c | 2 + clang/test/Driver/unique-source-file-names.c | 5 +++ llvm/lib/Transforms/Utils/ModuleUtils.cpp | 40 +++++++++---------- .../unique-source-file-names.ll | 22 ++++++++++ 10 files changed, 79 insertions(+), 21 deletions(-) create mode 100644 clang/test/CodeGen/unique-source-file-names.c create mode 100644 clang/test/Driver/unique-source-file-names.c create mode 100644 llvm/test/Transforms/ThinLTOBitcodeWriter/unique-source-file-names.ll diff --git a/clang/docs/ControlFlowIntegrity.rst b/clang/docs/ControlFlowIntegrity.rst index 2f2f8ccf4481b..baff9ab54ff26 100644 --- a/clang/docs/ControlFlowIntegrity.rst +++ b/clang/docs/ControlFlowIntegrity.rst @@ -42,6 +42,11 @@ default visibility setting is ``-fvisibility=default``, which would disable CFI checks for classes without visibility attributes. Most users will want to specify ``-fvisibility=hidden``, which enables CFI checks for such classes. +When using ``-fsanitize=cfi*`` with ``-flto=thin``, it is recommended +to reduce link times by passing `-funique-source-file-names +`_, provided +that your program is compatible with it. + Experimental support for :ref:`cross-DSO control flow integrity ` exists that does not require classes to have hidden LTO visibility. This cross-DSO support has unstable ABI at this time. diff --git a/clang/docs/UsersManual.rst b/clang/docs/UsersManual.rst index 2a93c2552d7dc..d4656a7e63c99 100644 --- a/clang/docs/UsersManual.rst +++ b/clang/docs/UsersManual.rst @@ -2297,6 +2297,16 @@ are listed below. pure ThinLTO, as all split regular LTO modules are merged and LTO linked with regular LTO. +.. option:: -f[no-]unique-source-file-names + + When enabled, allows the compiler to assume that each object file + passed to the linker has been compiled using a unique source file + name. This is useful for reducing link times when doing ThinLTO + in combination with whole-program devirtualization or CFI. + + A misuse of this flag will generally result in a duplicate symbol + error at link time. + .. option:: -fforce-emit-vtables In order to improve devirtualization, forces emitting of vtables even in diff --git a/clang/include/clang/Basic/CodeGenOptions.def b/clang/include/clang/Basic/CodeGenOptions.def index a436c0ec98d5b..c5990fb248689 100644 --- a/clang/include/clang/Basic/CodeGenOptions.def +++ b/clang/include/clang/Basic/CodeGenOptions.def @@ -278,6 +278,8 @@ CODEGENOPT(SanitizeCfiICallNormalizeIntegers, 1, 0) ///< Normalize integer types ///< CFI icall function signatures CODEGENOPT(SanitizeCfiCanonicalJumpTables, 1, 0) ///< Make jump table symbols canonical ///< instead of creating a local jump table. +CODEGENOPT(UniqueSourceFileNames, 1, 0) ///< Allow the compiler to assume that TUs + ///< have unique source file names at link time CODEGENOPT(SanitizeKcfiArity, 1, 0) ///< Embed arity in KCFI patchable function prefix CODEGENOPT(SanitizeCoverageType, 2, 0) ///< Type of sanitizer coverage ///< instrumentation. diff --git a/clang/include/clang/Driver/Options.td b/clang/include/clang/Driver/Options.td index c9d2bc5e81976..e9acb20348654 100644 --- a/clang/include/clang/Driver/Options.td +++ b/clang/include/clang/Driver/Options.td @@ -4140,6 +4140,13 @@ def ftrigraphs : Flag<["-"], "ftrigraphs">, Group, def fno_trigraphs : Flag<["-"], "fno-trigraphs">, Group, HelpText<"Do not process trigraph sequences">, Visibility<[ClangOption, CC1Option]>; +defm unique_source_file_names: BoolOption<"f", "unique-source-file-names", + CodeGenOpts<"UniqueSourceFileNames">, DefaultFalse, + PosFlag, + NegFlag, + BothFlags<[], [ClangOption], " the compiler to assume that each translation unit has a unique " + "source file name at link time">>, + Group; def funsigned_bitfields : Flag<["-"], "funsigned-bitfields">, Group; def funsigned_char : Flag<["-"], "funsigned-char">, Group; def fno_unsigned_char : Flag<["-"], "fno-unsigned-char">; diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp index 4a48c2f35ff23..26e09fe239242 100644 --- a/clang/lib/CodeGen/CodeGenModule.cpp +++ b/clang/lib/CodeGen/CodeGenModule.cpp @@ -1144,6 +1144,10 @@ void CodeGenModule::Release() { 1); } + if (CodeGenOpts.UniqueSourceFileNames) { + getModule().addModuleFlag(llvm::Module::Max, "Unique Source File Names", 1); + } + if (LangOpts.Sanitize.has(SanitizerKind::KCFI)) { getModule().addModuleFlag(llvm::Module::Override, "kcfi", 1); // KCFI assumes patchable-function-prefix is the same for all indirectly diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp index 65910e7fdaaa6..8506a5c00e7bc 100644 --- a/clang/lib/Driver/ToolChains/Clang.cpp +++ b/clang/lib/Driver/ToolChains/Clang.cpp @@ -7744,6 +7744,9 @@ void Clang::ConstructJob(Compilation &C, const JobAction &JA, Args.addOptInFlag(CmdArgs, options::OPT_fexperimental_late_parse_attributes, options::OPT_fno_experimental_late_parse_attributes); + Args.addOptInFlag(CmdArgs, options::OPT_funique_source_file_names, + options::OPT_fno_unique_source_file_names); + // Setup statistics file output. SmallString<128> StatsFile = getStatsFileName(Args, Output, Input, D); if (!StatsFile.empty()) { diff --git a/clang/test/CodeGen/unique-source-file-names.c b/clang/test/CodeGen/unique-source-file-names.c new file mode 100644 index 0000000000000..1d5a4a5e8e4c5 --- /dev/null +++ b/clang/test/CodeGen/unique-source-file-names.c @@ -0,0 +1,2 @@ +// RUN: %clang_cc1 -funique-source-file-names -triple x86_64-linux-gnu -emit-llvm %s -o - | FileCheck %s +// CHECK: !{i32 7, !"Unique Source File Names", i32 1} diff --git a/clang/test/Driver/unique-source-file-names.c b/clang/test/Driver/unique-source-file-names.c new file mode 100644 index 0000000000000..8322f0e37b0c7 --- /dev/null +++ b/clang/test/Driver/unique-source-file-names.c @@ -0,0 +1,5 @@ +// RUN: %clang -funique-source-file-names -### %s 2> %t +// RUN: FileCheck < %t %s + +// CHECK: "-cc1" +// CHECK: "-funique-source-file-names" diff --git a/llvm/lib/Transforms/Utils/ModuleUtils.cpp b/llvm/lib/Transforms/Utils/ModuleUtils.cpp index 1c31e851ef4b2..10efdd61d4553 100644 --- a/llvm/lib/Transforms/Utils/ModuleUtils.cpp +++ b/llvm/lib/Transforms/Utils/ModuleUtils.cpp @@ -345,27 +345,25 @@ void llvm::filterDeadComdatFunctions( std::string llvm::getUniqueModuleId(Module *M) { MD5 Md5; - bool ExportsSymbols = false; - auto AddGlobal = [&](GlobalValue &GV) { - if (GV.isDeclaration() || GV.getName().starts_with("llvm.") || - !GV.hasExternalLinkage() || GV.hasComdat()) - return; - ExportsSymbols = true; - Md5.update(GV.getName()); - Md5.update(ArrayRef{0}); - }; - - for (auto &F : *M) - AddGlobal(F); - for (auto &GV : M->globals()) - AddGlobal(GV); - for (auto &GA : M->aliases()) - AddGlobal(GA); - for (auto &IF : M->ifuncs()) - AddGlobal(IF); - - if (!ExportsSymbols) - return ""; + + auto *UniqueSourceFileNames = mdconst::extract_or_null( + M->getModuleFlag("Unique Source File Names")); + if (UniqueSourceFileNames && UniqueSourceFileNames->getZExtValue()) { + Md5.update(M->getSourceFileName()); + } else { + bool ExportsSymbols = false; + for (auto &GV : M->global_values()) { + if (GV.isDeclaration() || GV.getName().starts_with("llvm.") || + !GV.hasExternalLinkage() || GV.hasComdat()) + continue; + ExportsSymbols = true; + Md5.update(GV.getName()); + Md5.update(ArrayRef{0}); + } + + if (!ExportsSymbols) + return ""; + } MD5::MD5Result R; Md5.final(R); diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/unique-source-file-names.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/unique-source-file-names.ll new file mode 100644 index 0000000000000..0f3fd566f9b1c --- /dev/null +++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/unique-source-file-names.ll @@ -0,0 +1,22 @@ +; RUN: opt -thinlto-bc -thin-link-bitcode-file=%t2 -thinlto-split-lto-unit -o %t %s +; RUN: llvm-modextract -b -n 1 -o %t1 %t +; RUN: llvm-dis -o - %t1 | FileCheck %s + +source_filename = "unique-source-file-names.c" + +@llvm.global_ctors = appending global [1 x { i32, ptr, ptr }] [{ i32, ptr, ptr } { i32 65535, ptr @f, ptr null }] + +; CHECK: @g.45934e8a5251fb7adbecfff71a4e70ed = +@g = internal global i8 42, !type !0 + +declare void @sink(ptr) + +define internal void @f() { + call void @sink(ptr @g) + ret void +} + +!0 = !{i32 0, !"typeid"} + +!llvm.module.flags = !{!1} +!1 = !{i32 1, !"Unique Source File Names", i32 1} From d0c973a7a0149db3b71767d4c5a20a31e6a8ed5b Mon Sep 17 00:00:00 2001 From: Michael Spencer Date: Tue, 15 Apr 2025 11:19:07 -0700 Subject: [PATCH 08/63] [llvm][clang] Allocate a new stack instead of spawning a new thread to get more stack space (#133173) Clang spawns a new thread to avoid running out of stack space. This can make debugging and performance analysis more difficult as how the threads are connected is difficult to recover. This patch introduces `runOnNewStack` and applies it in Clang. On platforms that have good support for it this allocates a new stack and moves to it using assembly. Doing split stacks like this actually runs on most platforms, but many debuggers and unwinders reject the large or backwards stack offsets that occur. Apple platforms and tools are known to support this, so this only enables it there for now. --- clang/docs/ReleaseNotes.rst | 4 + clang/include/clang/Basic/Stack.h | 5 +- clang/lib/Basic/Stack.cpp | 40 ++---- clang/lib/Frontend/CompilerInstance.cpp | 2 +- .../llvm/Support/CrashRecoveryContext.h | 3 + llvm/include/llvm/Support/ProgramStack.h | 62 ++++++++++ llvm/lib/Support/CMakeLists.txt | 1 + llvm/lib/Support/CrashRecoveryContext.cpp | 11 ++ llvm/lib/Support/ProgramStack.cpp | 114 ++++++++++++++++++ llvm/unittests/Support/CMakeLists.txt | 1 + llvm/unittests/Support/ProgramStackTest.cpp | 35 ++++++ 11 files changed, 248 insertions(+), 30 deletions(-) create mode 100644 llvm/include/llvm/Support/ProgramStack.h create mode 100644 llvm/lib/Support/ProgramStack.cpp create mode 100644 llvm/unittests/Support/ProgramStackTest.cpp diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst index e63de32a0b2aa..22543821ee4b3 100644 --- a/clang/docs/ReleaseNotes.rst +++ b/clang/docs/ReleaseNotes.rst @@ -195,6 +195,10 @@ Non-comprehensive list of changes in this release - Added `__builtin_elementwise_exp10`. - For AMDPGU targets, added `__builtin_v_cvt_off_f32_i4` that maps to the `v_cvt_off_f32_i4` instruction. - Added `__builtin_elementwise_minnum` and `__builtin_elementwise_maxnum`. +- Clang itself now uses split stacks instead of threads for allocating more + stack space when running on Apple AArch64 based platforms. This means that + stack traces of Clang from debuggers, crashes, and profilers may look + different than before. New Compiler Flags ------------------ diff --git a/clang/include/clang/Basic/Stack.h b/clang/include/clang/Basic/Stack.h index 30ebd94aedd1f..9674b9d9b62c3 100644 --- a/clang/include/clang/Basic/Stack.h +++ b/clang/include/clang/Basic/Stack.h @@ -27,7 +27,10 @@ namespace clang { /// Call this once on each thread, as soon after starting the thread as /// feasible, to note the approximate address of the bottom of the stack. - void noteBottomOfStack(); + /// + /// \param ForceSet set to true if you know the call is near the bottom of a + /// new stack. Used for split stacks. + void noteBottomOfStack(bool ForceSet = false); /// Determine whether the stack is nearly exhausted. bool isStackNearlyExhausted(); diff --git a/clang/lib/Basic/Stack.cpp b/clang/lib/Basic/Stack.cpp index aa15d8e66950f..8cbb84943f8d3 100644 --- a/clang/lib/Basic/Stack.cpp +++ b/clang/lib/Basic/Stack.cpp @@ -13,33 +13,13 @@ #include "clang/Basic/Stack.h" #include "llvm/Support/CrashRecoveryContext.h" +#include "llvm/Support/ProgramStack.h" -#ifdef _MSC_VER -#include // for _AddressOfReturnAddress -#endif +static LLVM_THREAD_LOCAL uintptr_t BottomOfStack = 0; -static LLVM_THREAD_LOCAL void *BottomOfStack = nullptr; - -static void *getStackPointer() { -#if __GNUC__ || __has_builtin(__builtin_frame_address) - return __builtin_frame_address(0); -#elif defined(_MSC_VER) - return _AddressOfReturnAddress(); -#else - char CharOnStack = 0; - // The volatile store here is intended to escape the local variable, to - // prevent the compiler from optimizing CharOnStack into anything other - // than a char on the stack. - // - // Tested on: MSVC 2015 - 2019, GCC 4.9 - 9, Clang 3.2 - 9, ICC 13 - 19. - char *volatile Ptr = &CharOnStack; - return Ptr; -#endif -} - -void clang::noteBottomOfStack() { - if (!BottomOfStack) - BottomOfStack = getStackPointer(); +void clang::noteBottomOfStack(bool ForceSet) { + if (!BottomOfStack || ForceSet) + BottomOfStack = llvm::getStackPointer(); } bool clang::isStackNearlyExhausted() { @@ -51,7 +31,8 @@ bool clang::isStackNearlyExhausted() { if (!BottomOfStack) return false; - intptr_t StackDiff = (intptr_t)getStackPointer() - (intptr_t)BottomOfStack; + intptr_t StackDiff = + (intptr_t)llvm::getStackPointer() - (intptr_t)BottomOfStack; size_t StackUsage = (size_t)std::abs(StackDiff); // If the stack pointer has a surprising value, we do not understand this @@ -66,9 +47,12 @@ bool clang::isStackNearlyExhausted() { void clang::runWithSufficientStackSpaceSlow(llvm::function_ref Diag, llvm::function_ref Fn) { llvm::CrashRecoveryContext CRC; - CRC.RunSafelyOnThread([&] { - noteBottomOfStack(); + // Preserve the BottomOfStack in case RunSafelyOnNewStack uses split stacks. + uintptr_t PrevBottom = BottomOfStack; + CRC.RunSafelyOnNewStack([&] { + noteBottomOfStack(true); Diag(); Fn(); }, DesiredStackSize); + BottomOfStack = PrevBottom; } diff --git a/clang/lib/Frontend/CompilerInstance.cpp b/clang/lib/Frontend/CompilerInstance.cpp index 243e0a3c15b05..5fe80fc16482e 100644 --- a/clang/lib/Frontend/CompilerInstance.cpp +++ b/clang/lib/Frontend/CompilerInstance.cpp @@ -1265,7 +1265,7 @@ bool CompilerInstance::compileModule(SourceLocation ImportLoc, // Execute the action to actually build the module in-place. Use a separate // thread so that we get a stack large enough. - bool Crashed = !llvm::CrashRecoveryContext().RunSafelyOnThread( + bool Crashed = !llvm::CrashRecoveryContext().RunSafelyOnNewStack( [&]() { GenerateModuleFromModuleMapAction Action; Instance.ExecuteAction(Action); diff --git a/llvm/include/llvm/Support/CrashRecoveryContext.h b/llvm/include/llvm/Support/CrashRecoveryContext.h index 26ddf97b3ef02..31293d6715757 100644 --- a/llvm/include/llvm/Support/CrashRecoveryContext.h +++ b/llvm/include/llvm/Support/CrashRecoveryContext.h @@ -97,6 +97,9 @@ class CrashRecoveryContext { return RunSafelyOnThread([&]() { Fn(UserData); }, RequestedStackSize); } + bool RunSafelyOnNewStack(function_ref, + unsigned RequestedStackSize = 0); + /// Explicitly trigger a crash recovery in the current process, and /// return failure from RunSafely(). This function does not return. [[noreturn]] void HandleExit(int RetCode); diff --git a/llvm/include/llvm/Support/ProgramStack.h b/llvm/include/llvm/Support/ProgramStack.h new file mode 100644 index 0000000000000..3ce5de1c0e0d6 --- /dev/null +++ b/llvm/include/llvm/Support/ProgramStack.h @@ -0,0 +1,62 @@ +//===--- ProgramStack.h -----------------------------------------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_SUPPORT_PROGRAMSTACK_H +#define LLVM_SUPPORT_PROGRAMSTACK_H + +#include "llvm/ADT/STLFunctionalExtras.h" + +// LLVM_HAS_SPLIT_STACKS is exposed in the header because CrashRecoveryContext +// needs to know if it's running on another thread or not. +// +// Currently only Apple AArch64 is known to support split stacks in the debugger +// and other tooling. +#if defined(__APPLE__) && defined(__aarch64__) && \ + LLVM_HAS_CPP_ATTRIBUTE(gnu::naked) && __has_extension(gnu_asm) +# define LLVM_HAS_SPLIT_STACKS +# define LLVM_HAS_SPLIT_STACKS_AARCH64 +#endif + +namespace llvm { + +/// \returns an address close to the current value of the stack pointer. +/// +/// The value is not guaranteed to point to anything specific. It can be used to +/// estimate how much stack space has been used since the previous call. +uintptr_t getStackPointer(); + +/// \returns the default stack size for this platform. +/// +/// Based on \p RLIMIT_STACK or the equivalent. +unsigned getDefaultStackSize(); + +/// Runs Fn on a new stack of at least the given size. +/// +/// \param StackSize requested stack size. A size of 0 uses the default stack +/// size of the platform. +/// +/// The preferred implementation is split stacks on platforms that have a good +/// debugging experience for them. On other platforms a new thread is used. +void runOnNewStack(unsigned StackSize, function_ref Fn); + +template +R runOnNewStack(unsigned StackSize, function_ref Fn, Ts &&...Args) { + std::optional Ret; + runOnNewStack(StackSize, [&]() { Ret = Fn(std::forward(Args)...); }); + return std::move(*Ret); +} + +template +void runOnNewStack(unsigned StackSize, function_ref Fn, + Ts &&...Args) { + runOnNewStack(StackSize, [&]() { Fn(std::forward(Args)...); }); +} + +} // namespace llvm + +#endif // LLVM_SUPPORT_PROGRAMSTACK_H diff --git a/llvm/lib/Support/CMakeLists.txt b/llvm/lib/Support/CMakeLists.txt index 98ffd829d80b8..def37f3f278d0 100644 --- a/llvm/lib/Support/CMakeLists.txt +++ b/llvm/lib/Support/CMakeLists.txt @@ -295,6 +295,7 @@ add_llvm_component_library(LLVMSupport Path.cpp Process.cpp Program.cpp + ProgramStack.cpp RWMutex.cpp Signals.cpp Threading.cpp diff --git a/llvm/lib/Support/CrashRecoveryContext.cpp b/llvm/lib/Support/CrashRecoveryContext.cpp index f53aea177d612..88c38d7526e71 100644 --- a/llvm/lib/Support/CrashRecoveryContext.cpp +++ b/llvm/lib/Support/CrashRecoveryContext.cpp @@ -10,6 +10,7 @@ #include "llvm/Config/llvm-config.h" #include "llvm/Support/ErrorHandling.h" #include "llvm/Support/ExitCodes.h" +#include "llvm/Support/ProgramStack.h" #include "llvm/Support/Signals.h" #include "llvm/Support/thread.h" #include @@ -523,3 +524,13 @@ bool CrashRecoveryContext::RunSafelyOnThread(function_ref Fn, CRC->setSwitchedThread(); return Info.Result; } + +bool CrashRecoveryContext::RunSafelyOnNewStack(function_ref Fn, + unsigned RequestedStackSize) { +#ifdef LLVM_HAS_SPLIT_STACKS + return runOnNewStack(RequestedStackSize, + function_ref([&]() { return RunSafely(Fn); })); +#else + return RunSafelyOnThread(Fn, RequestedStackSize); +#endif +} diff --git a/llvm/lib/Support/ProgramStack.cpp b/llvm/lib/Support/ProgramStack.cpp new file mode 100644 index 0000000000000..9e5a546b34974 --- /dev/null +++ b/llvm/lib/Support/ProgramStack.cpp @@ -0,0 +1,114 @@ +//===--- RunOnNewStack.cpp - Crash Recovery -------------------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "llvm/Support/ProgramStack.h" +#include "llvm/Config/config.h" +#include "llvm/Support/Compiler.h" + +#ifdef LLVM_ON_UNIX +# include // for getrlimit +#endif + +#ifdef _MSC_VER +# include // for _AddressOfReturnAddress +#endif + +#ifndef LLVM_HAS_SPLIT_STACKS +# include "llvm/Support/thread.h" +#endif + +using namespace llvm; + +uintptr_t llvm::getStackPointer() { +#if __GNUC__ || __has_builtin(__builtin_frame_address) + return (uintptr_t)__builtin_frame_address(0); +#elif defined(_MSC_VER) + return (uintptr_t)_AddressOfReturnAddress(); +#else + volatile char CharOnStack = 0; + // The volatile store here is intended to escape the local variable, to + // prevent the compiler from optimizing CharOnStack into anything other + // than a char on the stack. + // + // Tested on: MSVC 2015 - 2019, GCC 4.9 - 9, Clang 3.2 - 9, ICC 13 - 19. + char *volatile Ptr = &CharOnStack; + return (uintptr_t)Ptr; +#endif +} + +unsigned llvm::getDefaultStackSize() { +#ifdef LLVM_ON_UNIX + rlimit RL; + getrlimit(RLIMIT_STACK, &RL); + return RL.rlim_cur; +#else + // Clang recursively parses, instantiates templates, and evaluates constant + // expressions. We've found 8MiB to be a reasonable stack size given the way + // Clang works and the way C++ is commonly written. + return 8 << 20; +#endif +} + +namespace { +#ifdef LLVM_HAS_SPLIT_STACKS_AARCH64 +[[gnu::naked]] void runOnNewStackImpl(void *Stack, void (*Fn)(void *), + void *Ctx) { + __asm__ volatile( + "mov x16, sp\n\t" + "sub x0, x0, #0x20\n\t" // subtract space from stack + "stp xzr, x16, [x0, #0x00]\n\t" // save old sp + "stp x29, x30, [x0, #0x10]\n\t" // save fp, lr + "mov sp, x0\n\t" // switch to new stack + "add x29, x0, #0x10\n\t" // switch to new frame + ".cfi_def_cfa w29, 16\n\t" + ".cfi_offset w30, -8\n\t" // lr + ".cfi_offset w29, -16\n\t" // fp + + "mov x0, x2\n\t" // Ctx is the only argument + "blr x1\n\t" // call Fn + + "ldp x29, x30, [sp, #0x10]\n\t" // restore fp, lr + "ldp xzr, x16, [sp, #0x00]\n\t" // load old sp + "mov sp, x16\n\t" + "ret" + ); +} +#endif + +#ifdef LLVM_HAS_SPLIT_STACKS +void callback(void *Ctx) { + (*reinterpret_cast *>(Ctx))(); +} +#endif +} // namespace + +#ifdef LLVM_HAS_SPLIT_STACKS +void llvm::runOnNewStack(unsigned StackSize, function_ref Fn) { + if (StackSize == 0) + StackSize = getDefaultStackSize(); + + // We use malloc here instead of mmap because: + // - it's simpler, + // - many malloc implementations will reuse the allocation in cases where + // we're bouncing accross the edge of a stack boundry, and + // - many malloc implemenations will already provide guard pages for + // allocations this large. + void *Stack = malloc(StackSize); + void *BottomOfStack = (char *)Stack + StackSize; + + runOnNewStackImpl(BottomOfStack, callback, &Fn); + + free(Stack); +} +#else +void llvm::runOnNewStack(unsigned StackSize, function_ref Fn) { + llvm::thread Thread( + StackSize == 0 ? std::nullopt : std::optional(StackSize), Fn); + Thread.join(); +} +#endif diff --git a/llvm/unittests/Support/CMakeLists.txt b/llvm/unittests/Support/CMakeLists.txt index 6c4e7cb689b20..e5bf820fb4d1c 100644 --- a/llvm/unittests/Support/CMakeLists.txt +++ b/llvm/unittests/Support/CMakeLists.txt @@ -70,6 +70,7 @@ add_llvm_unittest(SupportTests PerThreadBumpPtrAllocatorTest.cpp ProcessTest.cpp ProgramTest.cpp + ProgramStackTest.cpp RecyclerTest.cpp RegexTest.cpp ReverseIterationTest.cpp diff --git a/llvm/unittests/Support/ProgramStackTest.cpp b/llvm/unittests/Support/ProgramStackTest.cpp new file mode 100644 index 0000000000000..31dfb3b88ade6 --- /dev/null +++ b/llvm/unittests/Support/ProgramStackTest.cpp @@ -0,0 +1,35 @@ +//===- unittest/Support/ProgramStackTest.cpp ------------------------------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "llvm/Support/ProgramStack.h" +#include "llvm/Support/Process.h" +#include "gtest/gtest.h" + +using namespace llvm; + +static uintptr_t func(int &A) { + A = 7; + return getStackPointer(); +} + +static void func2(int &A) { + A = 5; +} + +TEST(ProgramStackTest, runOnNewStack) { + int A = 0; + uintptr_t Stack = runOnNewStack(0, function_ref(func), A); + EXPECT_EQ(A, 7); + intptr_t StackDiff = (intptr_t)llvm::getStackPointer() - (intptr_t)Stack; + size_t StackDistance = (size_t)std::abs(StackDiff); + // Page size is used as it's large enough to guarantee were not on the same + // stack but not too large to cause spurious failures. + EXPECT_GT(StackDistance, llvm::sys::Process::getPageSizeEstimate()); + runOnNewStack(0, function_ref(func2), A); + EXPECT_EQ(A, 5); +} From 429a84f8a4bf559f43f50072747ef49d3e3b2cf1 Mon Sep 17 00:00:00 2001 From: Michael Spencer Date: Tue, 15 Apr 2025 11:34:26 -0700 Subject: [PATCH 09/63] [clang] Fix ambiguity in `runOnNewStack` --- llvm/include/llvm/Support/ProgramStack.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/llvm/include/llvm/Support/ProgramStack.h b/llvm/include/llvm/Support/ProgramStack.h index 3ce5de1c0e0d6..232a7b5670b44 100644 --- a/llvm/include/llvm/Support/ProgramStack.h +++ b/llvm/include/llvm/Support/ProgramStack.h @@ -45,7 +45,8 @@ unsigned getDefaultStackSize(); void runOnNewStack(unsigned StackSize, function_ref Fn); template -R runOnNewStack(unsigned StackSize, function_ref Fn, Ts &&...Args) { +std::enable_if_t, R> +runOnNewStack(unsigned StackSize, function_ref Fn, Ts &&...Args) { std::optional Ret; runOnNewStack(StackSize, [&]() { Ret = Fn(std::forward(Args)...); }); return std::move(*Ret); From 227f4066befadf0d10eddb39947e35dbf820b1bb Mon Sep 17 00:00:00 2001 From: LLVM GN Syncbot Date: Tue, 15 Apr 2025 18:36:11 +0000 Subject: [PATCH 10/63] [gn build] Port d0c973a7a014 --- llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn | 1 + llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn | 1 + 2 files changed, 2 insertions(+) diff --git a/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn index 3a9f43b1070a7..0d2330cba6a7a 100644 --- a/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn +++ b/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn @@ -124,6 +124,7 @@ static_library("Support") { "Parallel.cpp", "PluginLoader.cpp", "PrettyStackTrace.cpp", + "ProgramStack.cpp", "RISCVAttributeParser.cpp", "RISCVAttributes.cpp", "RISCVISAUtils.cpp", diff --git a/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn b/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn index bf6a0b7523279..19418ad52147b 100644 --- a/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn +++ b/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn @@ -71,6 +71,7 @@ unittest("SupportTests") { "Path.cpp", "PerThreadBumpPtrAllocatorTest.cpp", "ProcessTest.cpp", + "ProgramStackTest.cpp", "ProgramTest.cpp", "RISCVAttributeParserTest.cpp", "RecyclerTest.cpp", From 13615f7b506a693783764da87dc80e97cf59b95c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Valentin=20Clement=20=28=E3=83=90=E3=83=AC=E3=83=B3?= =?UTF-8?q?=E3=82=BF=E3=82=A4=E3=83=B3=20=E3=82=AF=E3=83=AC=E3=83=A1?= =?UTF-8?q?=E3=83=B3=29?= Date: Tue, 15 Apr 2025 11:56:35 -0700 Subject: [PATCH 11/63] [flang][openacc] Allow if clause on atomic directives (#135451) The new version of the OpenACC specification will allow the if clause on the atomic directives. Allow it in `ACC.td` and update the parse node and parser in flang to support it. OpenACC dialect will need to be updated to support it as well. --- flang/include/flang/Parser/parse-tree.h | 10 +++--- flang/lib/Parser/openacc-parsers.cpp | 25 ++++++++------- .../Semantics/OpenACC/acc-atomic-validity.f90 | 32 +++++++++++++++++++ llvm/include/llvm/Frontend/OpenACC/ACC.td | 1 + 4 files changed, 53 insertions(+), 15 deletions(-) diff --git a/flang/include/flang/Parser/parse-tree.h b/flang/include/flang/Parser/parse-tree.h index eeb438991feee..0c2a5de3b71d2 100644 --- a/flang/include/flang/Parser/parse-tree.h +++ b/flang/include/flang/Parser/parse-tree.h @@ -5244,21 +5244,23 @@ EMPTY_CLASS(AccEndAtomic); // ACC ATOMIC READ struct AccAtomicRead { TUPLE_CLASS_BOILERPLATE(AccAtomicRead); - std::tuple, std::optional> + std::tuple, + std::optional> t; }; // ACC ATOMIC WRITE struct AccAtomicWrite { TUPLE_CLASS_BOILERPLATE(AccAtomicWrite); - std::tuple, std::optional> + std::tuple, + std::optional> t; }; // ACC ATOMIC UPDATE struct AccAtomicUpdate { TUPLE_CLASS_BOILERPLATE(AccAtomicUpdate); - std::tuple, Statement, + std::tuple, AccClauseList, Statement, std::optional> t; }; @@ -5268,7 +5270,7 @@ struct AccAtomicCapture { TUPLE_CLASS_BOILERPLATE(AccAtomicCapture); WRAPPER_CLASS(Stmt1, Statement); WRAPPER_CLASS(Stmt2, Statement); - std::tuple t; + std::tuple t; }; struct OpenACCAtomicConstruct { diff --git a/flang/lib/Parser/openacc-parsers.cpp b/flang/lib/Parser/openacc-parsers.cpp index fb731ee52cbba..072eba99826a1 100644 --- a/flang/lib/Parser/openacc-parsers.cpp +++ b/flang/lib/Parser/openacc-parsers.cpp @@ -187,22 +187,25 @@ TYPE_PARSER(construct( // 2.12 Atomic constructs TYPE_PARSER(construct(startAccLine >> "END ATOMIC"_tok)) -TYPE_PARSER("ATOMIC" >> - construct(verbatim("READ"_tok) / endAccLine, - statement(assignmentStmt), maybe(Parser{} / endAccLine))) +TYPE_PARSER("ATOMIC" >> construct(verbatim("READ"_tok), + Parser{} / endAccLine, + statement(assignmentStmt), + maybe(Parser{} / endAccLine))) -TYPE_PARSER("ATOMIC" >> - construct(verbatim("WRITE"_tok) / endAccLine, - statement(assignmentStmt), maybe(Parser{} / endAccLine))) +TYPE_PARSER("ATOMIC" >> construct(verbatim("WRITE"_tok), + Parser{} / endAccLine, + statement(assignmentStmt), + maybe(Parser{} / endAccLine))) TYPE_PARSER("ATOMIC" >> - construct(maybe(verbatim("UPDATE"_tok)) / endAccLine, - statement(assignmentStmt), maybe(Parser{} / endAccLine))) + construct(maybe(verbatim("UPDATE"_tok)), + Parser{} / endAccLine, statement(assignmentStmt), + maybe(Parser{} / endAccLine))) TYPE_PARSER("ATOMIC" >> - construct(verbatim("CAPTURE"_tok) / endAccLine, - statement(assignmentStmt), statement(assignmentStmt), - Parser{} / endAccLine)) + construct(verbatim("CAPTURE"_tok), + Parser{} / endAccLine, statement(assignmentStmt), + statement(assignmentStmt), Parser{} / endAccLine)) TYPE_PARSER( sourced(construct(Parser{})) || diff --git a/flang/test/Semantics/OpenACC/acc-atomic-validity.f90 b/flang/test/Semantics/OpenACC/acc-atomic-validity.f90 index ba68031b0f18b..07fb864695737 100644 --- a/flang/test/Semantics/OpenACC/acc-atomic-validity.f90 +++ b/flang/test/Semantics/OpenACC/acc-atomic-validity.f90 @@ -10,6 +10,7 @@ program openacc_atomic_validity integer :: i integer, parameter :: N = 256 integer, dimension(N) :: c + logical :: l !$acc parallel @@ -23,27 +24,58 @@ program openacc_atomic_validity !$acc atomic write c(i) = 10 + !$acc atomic write if(l) + c(i) = 10 + !$acc atomic write c(i) = 10 !$acc end atomic + !$acc atomic write if(.true.) + c(i) = 10 + !$acc end atomic + !$acc atomic read i = c(i) + + !$acc atomic read if(.true.) + i = c(i) !$acc atomic read i = c(i) !$acc end atomic + !$acc atomic read if(l) + i = c(i) + !$acc end atomic + + !ERROR: FINALIZE clause is not allowed on the ATOMIC READ FINALIZE IF(L) + !$acc atomic read finalize if(l) + i = c(i) + !$acc end atomic + !$acc atomic capture c(i) = i i = i + 1 !$acc end atomic + !$acc atomic capture if(l .EQV. .false.) + c(i) = i + i = i + 1 + !$acc end atomic + !$acc atomic update !ERROR: RHS of atomic update statement must be scalar !ERROR: LHS of atomic update statement must be scalar c = c + 1 + !$acc atomic update if(i == 0) + c(i) = c(i) + 1 + + !ERROR: At most one IF clause can appear on the ATOMIC UPDATE IF(I == 0) IF(.TRUE.) + !$acc atomic update if(i == 0) if(.true.) + c(i) = c(i) + 1 + !$acc end parallel end program openacc_atomic_validity diff --git a/llvm/include/llvm/Frontend/OpenACC/ACC.td b/llvm/include/llvm/Frontend/OpenACC/ACC.td index 2acee9bcc7195..e1a4183785d1f 100644 --- a/llvm/include/llvm/Frontend/OpenACC/ACC.td +++ b/llvm/include/llvm/Frontend/OpenACC/ACC.td @@ -270,6 +270,7 @@ def ACCC_Unknown : Clause<"unknown"> { // 2.12 def ACC_Atomic : Directive<"atomic"> { + let allowedOnceClauses = [VersionedClause]; let association = AS_Block; let category = CA_Executable; } From 8f25e43055058a6a16bf44573feb37a9ce51dc1a Mon Sep 17 00:00:00 2001 From: AdityaK Date: Tue, 15 Apr 2025 12:11:24 -0700 Subject: [PATCH 12/63] [NFC] Rename hasSameElementsOrSplat to hasSameNumElementsOrSplat (#133183) Makes it less confusing as this function only matches the number of elements --- mlir/lib/IR/BuiltinAttributes.cpp | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp index daf79dc5de981..67d1ad927cacc 100644 --- a/mlir/lib/IR/BuiltinAttributes.cpp +++ b/mlir/lib/IR/BuiltinAttributes.cpp @@ -589,7 +589,7 @@ static APInt readBits(const char *rawData, size_t bitPos, size_t bitWidth) { /// Returns true if 'values' corresponds to a splat, i.e. one element, or has /// the same element count as 'type'. template -static bool hasSameElementsOrSplat(ShapedType type, const Values &values) { +static bool hasSameNumElementsOrSplat(ShapedType type, const Values &values) { return (values.size() == 1) || (type.getNumElements() == static_cast(values.size())); } @@ -901,7 +901,7 @@ bool DenseElementsAttr::classof(Attribute attr) { DenseElementsAttr DenseElementsAttr::get(ShapedType type, ArrayRef values) { - assert(hasSameElementsOrSplat(type, values)); + assert(hasSameNumElementsOrSplat(type, values)); Type eltType = type.getElementType(); @@ -985,7 +985,7 @@ DenseElementsAttr DenseElementsAttr::get(ShapedType type, DenseElementsAttr DenseElementsAttr::get(ShapedType type, ArrayRef values) { - assert(hasSameElementsOrSplat(type, values)); + assert(hasSameNumElementsOrSplat(type, values)); assert(type.getElementType().isInteger(1)); std::vector buff(llvm::divideCeil(values.size(), CHAR_BIT)); @@ -1020,7 +1020,7 @@ DenseElementsAttr DenseElementsAttr::get(ShapedType type, DenseElementsAttr DenseElementsAttr::get(ShapedType type, ArrayRef values) { assert(type.getElementType().isIntOrIndex()); - assert(hasSameElementsOrSplat(type, values)); + assert(hasSameNumElementsOrSplat(type, values)); size_t storageBitWidth = getDenseElementStorageWidth(type.getElementType()); return DenseIntOrFPElementsAttr::getRaw(type, storageBitWidth, values); } @@ -1028,7 +1028,7 @@ DenseElementsAttr DenseElementsAttr::get(ShapedType type, ArrayRef> values) { ComplexType complex = llvm::cast(type.getElementType()); assert(llvm::isa(complex.getElementType())); - assert(hasSameElementsOrSplat(type, values)); + assert(hasSameNumElementsOrSplat(type, values)); size_t storageBitWidth = getDenseElementStorageWidth(complex) / 2; ArrayRef intVals(reinterpret_cast(values.data()), values.size() * 2); @@ -1041,7 +1041,7 @@ DenseElementsAttr DenseElementsAttr::get(ShapedType type, DenseElementsAttr DenseElementsAttr::get(ShapedType type, ArrayRef values) { assert(llvm::isa(type.getElementType())); - assert(hasSameElementsOrSplat(type, values)); + assert(hasSameNumElementsOrSplat(type, values)); size_t storageBitWidth = getDenseElementStorageWidth(type.getElementType()); return DenseIntOrFPElementsAttr::getRaw(type, storageBitWidth, values); } @@ -1050,7 +1050,7 @@ DenseElementsAttr::get(ShapedType type, ArrayRef> values) { ComplexType complex = llvm::cast(type.getElementType()); assert(llvm::isa(complex.getElementType())); - assert(hasSameElementsOrSplat(type, values)); + assert(hasSameNumElementsOrSplat(type, values)); ArrayRef apVals(reinterpret_cast(values.data()), values.size() * 2); size_t storageBitWidth = getDenseElementStorageWidth(complex) / 2; From 9a1ece26126363c64c67d9a6e357076e814acf9e Mon Sep 17 00:00:00 2001 From: marius doerner Date: Tue, 15 Apr 2025 21:13:56 +0200 Subject: [PATCH 13/63] [clang] Clear `NeedsCleaning` flag after `ExpandBuiltinMacro` (#133574) After builtin macro expansion in `Preprocessor::ExpandBuiltinMacro` the result token may have the `Token::NeedsCleaning` flag set which causes an assertion failure later on when the lexer retrieves the spelling of the token in `getSpellingSlow`. This commit adds an `Tok.clearFlag(Token::NeedsCleaning)` call to the end of `ExpandBuiltinMacro`. Closes #128384 --- clang/docs/ReleaseNotes.rst | 2 + clang/lib/Lex/PPMacroExpansion.cpp | 1 + clang/test/Preprocessor/embed___has_embed.c | 19 +++ clang/test/Preprocessor/has_attribute.c | 20 ++++ clang/test/Preprocessor/has_attribute.cpp | 20 ++++ clang/test/Preprocessor/has_c_attribute.c | 20 ++++ clang/test/Preprocessor/has_include.c | 49 ++++++++ clang/test/Preprocessor/pr133574.c | 121 ++++++++++++++++++++ 8 files changed, 252 insertions(+) create mode 100644 clang/test/Preprocessor/pr133574.c diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst index 22543821ee4b3..6025e76029d19 100644 --- a/clang/docs/ReleaseNotes.rst +++ b/clang/docs/ReleaseNotes.rst @@ -404,6 +404,8 @@ Bug Fixes in This Version - Defining an integer literal suffix (e.g., ``LL``) before including ```` in a freestanding build no longer causes invalid token pasting when using the ``INTn_C`` macros. (#GH85995) +- Fixed an assertion failure in the expansion of builtin macros like ``__has_embed()`` with line breaks before the + closing paren. (#GH133574) - Clang no longer accepts invalid integer constants which are too large to fit into any (standard or extended) integer type when the constant is unevaluated. Merely forming the token is sufficient to render the program invalid. Code diff --git a/clang/lib/Lex/PPMacroExpansion.cpp b/clang/lib/Lex/PPMacroExpansion.cpp index 8e35d56d3f1a6..37ac1bf07e9c0 100644 --- a/clang/lib/Lex/PPMacroExpansion.cpp +++ b/clang/lib/Lex/PPMacroExpansion.cpp @@ -2089,6 +2089,7 @@ void Preprocessor::ExpandBuiltinMacro(Token &Tok) { CreateString(OS.str(), Tok, Tok.getLocation(), Tok.getLocation()); Tok.setFlagValue(Token::StartOfLine, IsAtStartOfLine); Tok.setFlagValue(Token::LeadingSpace, HasLeadingSpace); + Tok.clearFlag(Token::NeedsCleaning); } void Preprocessor::markMacroAsUsed(MacroInfo *MI) { diff --git a/clang/test/Preprocessor/embed___has_embed.c b/clang/test/Preprocessor/embed___has_embed.c index 43a3068b5f53a..2705b5ef6fd5b 100644 --- a/clang/test/Preprocessor/embed___has_embed.c +++ b/clang/test/Preprocessor/embed___has_embed.c @@ -58,3 +58,22 @@ unsigned char buffer[] = { #else #error 17 #endif + +#if __has_embed(__FILE__\ +) +#else +#error 18 +#endif + +#define F __FI\ +LE__ +#if __has_embed(F) +#else +#error 19 +#endif + +#if __has_embed(F\ +) +#else +#error 20 +#endif diff --git a/clang/test/Preprocessor/has_attribute.c b/clang/test/Preprocessor/has_attribute.c index 0ba664a53e649..6f6f519bd299d 100644 --- a/clang/test/Preprocessor/has_attribute.c +++ b/clang/test/Preprocessor/has_attribute.c @@ -68,3 +68,23 @@ int has_no_volatile_attribute(); int has_fallthrough; #endif // CHECK: int has_fallthrough; + +#if __has_attribute(F\ +) +int has_fallthrough_2; +#endif +// CHECK: int has_fallthrough_2; + +#define F_2 fall\ +through + +#if __has_attribute(F_2) +int has_fallthrough_3; +#endif +// CHECK: int has_fallthrough_3; + +#if __has_attribute(F_2\ +) +int has_fallthrough_4; +#endif +// CHECK: int has_fallthrough_4; diff --git a/clang/test/Preprocessor/has_attribute.cpp b/clang/test/Preprocessor/has_attribute.cpp index 00ec57615c84b..72af6de27e8bb 100644 --- a/clang/test/Preprocessor/has_attribute.cpp +++ b/clang/test/Preprocessor/has_attribute.cpp @@ -116,6 +116,26 @@ int funclike_1; int funclike_2; #endif // CHECK: int funclike_2; + +#if __has_cpp_attribute(CF\ +) +int has_clang_falthrough_5; +#endif +// CHECK: int has_clang_falthrough_5; + +#define CF_2 clang::\ +fallthrough + +#if __has_cpp_attribute(CF_2) +int has_clang_falthrough_6; +#endif +// CHECK: int has_clang_falthrough_6; + +#if __has_cpp_attribute(CF_2\ +) +int has_clang_falthrough_7; +#endif +// CHECK: int has_clang_falthrough_7; } // Test for Microsoft __declspec attributes diff --git a/clang/test/Preprocessor/has_c_attribute.c b/clang/test/Preprocessor/has_c_attribute.c index 3332571d758c8..d8be13f5898a9 100644 --- a/clang/test/Preprocessor/has_c_attribute.c +++ b/clang/test/Preprocessor/has_c_attribute.c @@ -84,3 +84,23 @@ int funclike_1; int funclike_2; #endif // CHECK: int funclike_2; + +#if __has_c_attribute(CL\ +) +int has_clang_likely_5; +#endif +// CHECK: int has_clang_likely_5; + +#define CL_2 clang::\ +likely + +#if __has_c_attribute(CL_2) +int has_clang_likely_6; +#endif +// CHECK: int has_clang_likely_6; + +#if __has_c_attribute(CL_2\ +) +int has_clang_likely_7; +#endif +// CHECK: int has_clang_likely_7; diff --git a/clang/test/Preprocessor/has_include.c b/clang/test/Preprocessor/has_include.c index c95025d83860a..ff199bf23063f 100644 --- a/clang/test/Preprocessor/has_include.c +++ b/clang/test/Preprocessor/has_include.c @@ -197,3 +197,52 @@ __has_include #ifdef FOO #elif __has_include() #endif + +#if __has_include(\ +) +#else + #error "__has_include failed (10)." +#endif + +#define MACRO6 +#if __has_include(MACRO6\ +) +#else + #error "__has_include failed (11)." +#endif + +#if __has_include_next(/*expected-warning {{#include_next in primary source file}}*/\ +) +#else + #error "__has_include_next failed (9)." +#endif + +#if __has_include_next(MACRO6/*expected-warning {{#include_next in primary source file}}*/\ +) +#else + #error "__has_include_next failed (10)." +#endif + +#define MACRO7 +#if __has_include(MACRO7) +#else + #error "__has_include failed (12)." +#endif + +#if __has_include(MACRO7\ +) +#else + #error "__has_include failed (13)." +#endif + +#if __has_include_next(MACRO7) //expected-warning {{#include_next in primary source file}} +#else + #error "__has_include_next failed (11)." +#endif + +#if __has_include_next(MACRO7/*expected-warning {{#include_next in primary source file}}*/\ +) +#else + #error "__has_include_next failed (12)." +#endif diff --git a/clang/test/Preprocessor/pr133574.c b/clang/test/Preprocessor/pr133574.c new file mode 100644 index 0000000000000..a34073e63b760 --- /dev/null +++ b/clang/test/Preprocessor/pr133574.c @@ -0,0 +1,121 @@ +// RUN: %clang_cc1 -E -verify %s +// expected-no-diagnostics + +#define DATE_LBR __D\ +ATE__ + +const char* test1(void) { + return __DATE\ +__; +} +const char* test2(void) { + return DATE_LBR; +} + +#define TIME_LBR __TIME_\ +_ + +const char* test3(void) { + return __TIM\ +E__; +} + +const char* test4(void) { + return TIME_LBR; +} + +#define LINE_LBR __LI\ +NE__ + +int test5(void) { + return _\ +_LINE__; +} + +int test6(void) { + return LINE_LBR; +} + +#define FILE_LBR __FI\ +LE__ + +const char* test7(void) { + return __\ +FILE__; +} + +const char* test8(void) { + return FILE_LBR; +} + +#define FILE_NAME_LBR __FILE_NA\ +ME__ + +const char* test9(void) { + return __FILE_NAM\ +E__; +} + +const char* test10(void) { + return FILE_NAME_LBR; +} + +#define BASE_FILE_LBR __BASE_FIL\ +E__ + +const char* test11(void) { + return __BASE_\ +FILE__; +} + +const char* test12(void) { + return BASE_FILE_LBR; +} + +#define INCLUDE_LEVEL_LBR __INCLUDE\ +_LEVEL__ + +int test13(void) { + return __IN\ +CLUDE_LEVEL__; +} + +int test14(void) { + return INCLUDE_LEVEL_LBR; +} + +#define TIMESTAMP_LBR __TIMESTA\ +MP__ + +const char* test15(void) { + return __TIMESTA\ +MP__; +} + +const char* test16(void) { + return TIMESTAMP_LBR; +} + +#define FLT_EVAL_METHOD_LBR __FLT_EVAL_METH\ +OD__ + +int test17(void) { + return __FL\ +T_EVAL_METHOD__; +} + +int test18(void) { + return FLT_EVAL_METHOD_LBR; +} + +#define COUNTER_LBR __COUNTE\ +R__ + +int test19(void) { + return _\ +_COUNTER__; +} + +int test20(void) { + return COUNTER_LBR; +} From b581bd3429b28420ff473f700fe96c18127a475d Mon Sep 17 00:00:00 2001 From: Scott Manley Date: Tue, 15 Apr 2025 14:21:19 -0500 Subject: [PATCH 14/63] [flang][OpenACC] use correct type when create private box init recipe (#135698) The recipe for initializing private box types was incorrect because hlfir::createTempFromMold() is not a suitable utility function when the box element type is a trivial type. --- flang/lib/Lower/OpenACC.cpp | 24 ++++++++++++------- flang/test/Lower/OpenACC/acc-private.f90 | 30 ++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 8 deletions(-) diff --git a/flang/lib/Lower/OpenACC.cpp b/flang/lib/Lower/OpenACC.cpp index 3dd35ed9ae481..c83e277b996f3 100644 --- a/flang/lib/Lower/OpenACC.cpp +++ b/flang/lib/Lower/OpenACC.cpp @@ -522,13 +522,17 @@ static void genPrivateLikeInitRegion(mlir::OpBuilder &builder, RecipeOp recipe, mlir::Type ty, mlir::Location loc) { mlir::Value retVal = recipe.getInitRegion().front().getArgument(0); ty = fir::unwrapRefType(ty); - if (fir::isa_trivial(ty)) { + + auto getDeclareOpForType = [&](mlir::Type ty) -> hlfir::DeclareOp { auto alloca = builder.create(loc, ty); - auto declareOp = builder.create( + return builder.create( loc, alloca, accPrivateInitName, /*shape=*/nullptr, llvm::ArrayRef{}, /*dummy_scope=*/nullptr, fir::FortranVariableFlagsAttr{}); - retVal = declareOp.getBase(); + }; + + if (fir::isa_trivial(ty)) { + retVal = getDeclareOpForType(ty).getBase(); } else if (auto seqTy = mlir::dyn_cast_or_null(ty)) { if (fir::isa_trivial(seqTy.getEleTy())) { mlir::Value shape; @@ -552,12 +556,16 @@ static void genPrivateLikeInitRegion(mlir::OpBuilder &builder, RecipeOp recipe, } } else if (auto boxTy = mlir::dyn_cast_or_null(ty)) { mlir::Type innerTy = fir::unwrapRefType(boxTy.getEleTy()); - if (!fir::isa_trivial(innerTy) && !mlir::isa(innerTy)) + if (fir::isa_trivial(innerTy)) { + retVal = getDeclareOpForType(ty).getBase(); + } else if (mlir::isa(innerTy)) { + fir::FirOpBuilder firBuilder{builder, recipe.getOperation()}; + hlfir::Entity source = hlfir::Entity{retVal}; + auto [temp, cleanup] = hlfir::createTempFromMold(loc, firBuilder, source); + retVal = temp; + } else { TODO(loc, "Unsupported boxed type in OpenACC privatization"); - fir::FirOpBuilder firBuilder{builder, recipe.getOperation()}; - hlfir::Entity source = hlfir::Entity{retVal}; - auto [temp, cleanup] = hlfir::createTempFromMold(loc, firBuilder, source); - retVal = temp; + } } builder.create(loc, retVal); } diff --git a/flang/test/Lower/OpenACC/acc-private.f90 b/flang/test/Lower/OpenACC/acc-private.f90 index 50c7a258bb567..356bb9d825d8e 100644 --- a/flang/test/Lower/OpenACC/acc-private.f90 +++ b/flang/test/Lower/OpenACC/acc-private.f90 @@ -87,6 +87,13 @@ ! CHECK: acc.yield %[[DECLARE]]#0 : !fir.box> ! CHECK: } +! CHECK-LABEL: @privatization_ref_box_heap_i32 : !fir.ref>> init { +! CHECK: ^bb0(%arg0: !fir.ref>>): +! CHECK: %[[ALLOCA:.*]] = fir.alloca !fir.box> +! CHECK: %[[DECLARE:.*]]:2 = hlfir.declare %[[ALLOCA]] {uniq_name = "acc.private.init"} : (!fir.ref>>) -> (!fir.ref>>, !fir.ref>>) +! CHECK: acc.yield %[[DECLARE]]#0 : !fir.ref>> +! CHECK: } + ! CHECK-LABEL: acc.private.recipe @privatization_ref_box_heap_Uxi32 : !fir.ref>>> init { ! CHECK: ^bb0(%[[ARG0:.*]]: !fir.ref>>>): ! CHECK: %[[LOADBOX:.*]] = fir.load %[[ARG0]] : !fir.ref>>> @@ -292,6 +299,29 @@ subroutine acc_private_allocatable_array(a, n) ! CHECK: acc.loop {{.*}} private({{.*}}@privatization_ref_box_heap_Uxi32 -> %[[PRIVATE]] : !fir.ref>>>) ! CHECK: acc.serial private(@privatization_ref_box_heap_Uxi32 -> %{{.*}} : !fir.ref>>>) +subroutine acc_private_allocatable_scalar(b, a, n) + integer :: a(n) + integer, allocatable :: b + integer :: i, n + + !$acc parallel loop private(b) + do i = 1, n + a(i) = b + end do + + !$acc serial private(b) + a(i) = b + !$acc end serial +end subroutine + +! CHECK-LABEL: func.func @_QPacc_private_allocatable_scalar( +! CHECK-SAME: %[[ARG0:.*]]: !fir.ref>> {fir.bindc_name = "b"} +! CHECK: %[[DECLA_B:.*]]:2 = hlfir.declare %arg0 dummy_scope %0 {fortran_attrs = #fir.var_attrs, uniq_name = "_QFacc_private_allocatable_scalarEb"} : (!fir.ref>>, !fir.dscope) -> (!fir.ref>>, !fir.ref>>) +! CHECK: acc.parallel {{.*}} { +! CHECK: %[[PRIVATE:.*]] = acc.private varPtr(%[[DECLA_B]]#0 : !fir.ref>>) -> !fir.ref>> {name = "b"} +! CHECK: acc.loop {{.*}} private({{.*}}@privatization_ref_box_heap_i32 -> %[[PRIVATE]] : !fir.ref>>) +! CHECK: acc.serial private(@privatization_ref_box_heap_i32 -> %{{.*}} : !fir.ref>>) { + subroutine acc_private_pointer_array(a, n) integer, pointer :: a(:) integer :: i, n From bd9c5112c750f07df2886562b227d9e2aeb338c8 Mon Sep 17 00:00:00 2001 From: Tai Ly Date: Tue, 15 Apr 2025 14:36:42 -0500 Subject: [PATCH 15/63] [mlir][tosa] Add error_if checks for Transpose (#135219) This adds missing error_if checking for Transpose Op also moved all transpose op's verifier tests from invalid.mlir to verifier.mlir Signed-off-by: Tai Ly --- mlir/lib/Dialect/Tosa/IR/TosaOps.cpp | 48 ++++++---- mlir/test/Dialect/Tosa/invalid.mlir | 112 ------------------------ mlir/test/Dialect/Tosa/verifier.mlir | 126 +++++++++++++++++++++++++++ 3 files changed, 155 insertions(+), 131 deletions(-) create mode 100644 mlir/test/Dialect/Tosa/verifier.mlir diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp index 9579d71a2afe9..bce5b226635f3 100644 --- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp +++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp @@ -1981,23 +1981,28 @@ LogicalResult tosa::TransposeOp::verify() { .failed()) { return failure(); } - TensorType inputType = getInput1().getType(); - TensorType outputType = getOutput().getType(); + + const ShapeAdaptor inputShape(getInput1().getType()); + const ShapeAdaptor outputShape(getOutput().getType()); + const llvm::ArrayRef constantPerms = getPerms(); - if (inputType.hasRank() && - constantPerms.size() != static_cast(inputType.getRank())) + if (inputShape.hasRank() && + constantPerms.size() != static_cast(inputShape.getRank())) return emitOpError() << "expected perms attribute to have size " - << inputType.getRank() << " (input rank) but got size " + << inputShape.getRank() + << " (input rank) but got size " << constantPerms.size(); - if (inputType.hasRank() && outputType.hasRank() && - inputType.getRank() != outputType.getRank()) + + if (inputShape.hasRank() && outputShape.hasRank() && + inputShape.getRank() != outputShape.getRank()) return emitOpError() << "expected input tensor rank to equal result tensor rank"; - if (outputType.hasRank() && - constantPerms.size() != static_cast(outputType.getRank())) + + if (outputShape.hasRank() && + constantPerms.size() != static_cast(outputShape.getRank())) return emitOpError() << "expected perms attribute to have size " - << outputType.getRank() + << outputShape.getRank() << " (output rank) but got size " << constantPerms.size(); @@ -2010,22 +2015,27 @@ LogicalResult tosa::TransposeOp::verify() { constantPerms, [](int32_t v) -> int64_t { return v; })))) return emitOpError() << "expected valid permutation indices"; + // ERROR_IF(tensor_size(shape1) != tensor_size(shape)) + if (inputShape.hasStaticShape() && outputShape.hasStaticShape() && + inputShape.getNumElements() != outputShape.getNumElements()) + return emitOpError() << "expected input1 and output to have same numbers " + "of elements, got " + << inputShape.getNumElements() << " and " + << outputShape.getNumElements(); + // Verify that the types of the input and output tensors are properly // permuted. - if (inputType.hasRank() && outputType.hasRank()) { - assert(constantPerms.size() == static_cast(inputType.getRank()) && - inputType.getRank() == outputType.getRank()); - - for (auto i = 0; i < outputType.getRank(); i++) { - if (inputType.isDynamicDim(constantPerms[i]) || - outputType.isDynamicDim(i)) + if (inputShape.hasRank() && outputShape.hasRank()) { + for (auto i = 0; i < outputShape.getRank(); i++) { + if (inputShape.isDynamicDim(constantPerms[i]) || + outputShape.isDynamicDim(i)) continue; - if (inputType.getDimSize(constantPerms[i]) != outputType.getDimSize(i)) + if (inputShape.getDimSize(constantPerms[i]) != outputShape.getDimSize(i)) return emitOpError() << "expected output tensor dim " << i << " to match " << "input dim " << constantPerms[i] << " with value of " - << inputType.getDimSize(constantPerms[i]); + << inputShape.getDimSize(constantPerms[i]); } } diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir index 8d8f4f562c4b6..c0b251885de5c 100644 --- a/mlir/test/Dialect/Tosa/invalid.mlir +++ b/mlir/test/Dialect/Tosa/invalid.mlir @@ -368,79 +368,6 @@ func.func @test_pad_padding_shape_mismatch(%arg0: tensor<13x21x3xf32>) -> tensor // ----- -func.func @test_transpose_io_rank_mismatch(%arg0: tensor<13x21x3xf32>, %arg1: tensor<3xi32>) -> tensor<3x13x21x1xf32> { - // expected-error@+1 {{'tosa.transpose' op expected input tensor rank to equal result tensor rank}} - %0 = tosa.transpose %arg0 {perms = array}: (tensor<13x21x3xf32>) -> tensor<3x13x21x1xf32> - return %0 : tensor<3x13x21x1xf32> -} - -// ----- - -func.func @test_transpose_rank0_perms() { - %14 = tensor.empty() : tensor<5x27xi64> - // expected-error@+1 {{'tosa.transpose' op expected perms attribute to have size 2 (input rank) but got size 0}} - %72 = tosa.transpose %14 {perms = array }: (tensor<5x27xi64>) -> tensor - return -} - -// ----- - -func.func @test_transpose_invalid_perms_size(%arg0: tensor<13x21x3xf32>) -> tensor<3x13x21xf32> { - // expected-error@+1 {{'tosa.transpose' op expected perms attribute to have size 3 (input rank) but got size 7}} - %0 = tosa.transpose %arg0 {perms = array }: (tensor<13x21x3xf32>) -> tensor<3x13x21xf32> - return %0 : tensor<3x13x21xf32> -} - -// ----- - -func.func @test_transpose_invalid_permutation_tensor(%arg0: tensor<13x21x3xf32>) -> tensor { - // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} - %0 = tosa.transpose %arg0 {perms = array }: (tensor<13x21x3xf32>) -> tensor - return %0 : tensor -} - -// ----- - -func.func @test_transpose_invalid_permutation_negative(%arg0: tensor<3x2xi32>) -> tensor<*xi32> { - // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} - %1 = tosa.transpose %arg0 {perms = array }: (tensor<3x2xi32>) -> tensor<*xi32> - return %1 : tensor<*xi32> -} - -// ----- - -func.func @test_transpose_invalid_permutation_tensor_above_range(%arg0: tensor<3x2xi32>) -> tensor<*xi32> { - // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} - %1 = tosa.transpose %arg0 {perms = array }: (tensor<3x2xi32>) -> tensor<*xi32> - return %1 : tensor<*xi32> -} - -// ----- - -func.func @test_transpose_invalid_permutation_types(%arg0: tensor<3x2xi32>) -> tensor<3x4xi32> { - // expected-error@+1 {{'tosa.transpose' op expected output tensor dim 0 to match input dim 1 with value of 2}} - %1 = tosa.transpose %arg0 {perms = array }: (tensor<3x2xi32>) -> tensor<3x4xi32> - return %1 : tensor<3x4xi32> -} - -// ----- - -func.func @test_transpose_invalid_permutation_types_dynamic_dim_ok(%arg0: tensor<2x?xi32>) -> tensor<3x4xi32> { - // expected-error@+1 {{'tosa.transpose' op expected output tensor dim 1 to match input dim 0 with value of 2}} - %1 = tosa.transpose %arg0 {perms = array }: (tensor<2x?xi32>) -> tensor<3x4xi32> - return %1 : tensor<3x4xi32> -} - -// ----- - -func.func @test_transpose_element_type_mismatch(%arg0: tensor<2x3xi32>) -> tensor<3x2xf32> { - // expected-error@+1 {{'tosa.transpose' op failed to verify that all of {input1, output} have same element type}} - %1 = tosa.transpose %arg0 {perms = array} : (tensor<2x3xi32>) -> tensor<3x2xf32> - return %1 : tensor<3x2xf32> -} - -// ----- - func.func @test_reduce_sum_type_mismatch(%arg0 : tensor<2x3x4x5xf32>) -> () { // expected-error@+2 {{failed to infer returned types}} // expected-error@+1 {{'tosa.reduce_sum' op inferred type(s) 'tensor<1x3x4x5xf32>' are incompatible with return type(s) of operation 'tensor<1x3x4x5xi32>'}} @@ -783,37 +710,6 @@ func.func @test_tile_io_rank_mismatch() { return } -// ----- - -// CHECK-LABEL: @test_invalid_constant_permutation -func.func @test_invalid_constant_permutation() { - %0 = tensor.empty() : tensor<3x4x5xi32> - // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} - %2 = tosa.transpose %0 {perms = array}: (tensor<3x4x5xi32>) -> tensor<3x4x5xi32> - return -} - -// ----- - -// CHECK-LABEL: test_rank_size_constant_permutation -func.func @test_rank_size_constant_permutation() { - %0 = arith.constant 6 : index - %2 = tensor.empty(%0) : tensor - // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} - %3 = tosa.transpose %2 {perms = array}: (tensor) -> tensor - return -} - -// ----- - -// CHECK-LABEL: test_large_constant_permutation -func.func @test_large_constant_permutation() { - %0 = arith.constant 6 : index - %2 = tensor.empty(%0) : tensor - // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} - %3 = tosa.transpose %2 {perms = array}: (tensor) -> tensor - return -} // ----- @@ -2061,14 +1957,6 @@ func.func @test_scalar_tile(%arg0: tensor) -> tensor<*xf32> { // ----- -func.func @test_scalar_output_transpose(%arg0: tensor<*xf32>) -> tensor { - // expected-error@+1 {{'tosa.transpose' op result #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor'}} - %1 = tosa.transpose %arg0 {perms = array} : (tensor<*xf32>) -> tensor - return %1 : tensor -} - -// ----- - // CHECK-LABEL: test_add_i1 func.func @test_add_i1(%arg0: tensor<13x21x1xi1>, %arg1: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> { // expected-error@+1 {{'tosa.add' op illegal: operand/result data types not supported}} diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir new file mode 100644 index 0000000000000..c49cbecd25c78 --- /dev/null +++ b/mlir/test/Dialect/Tosa/verifier.mlir @@ -0,0 +1,126 @@ +//-------------------------------------------------------------------------------------------------- +// Test expected errors generated by verifier checks. +//-------------------------------------------------------------------------------------------------- + +// RUN: mlir-opt %s -split-input-file -verify-diagnostics + +// ----- + +func.func @test_transpose_io_rank_mismatch(%arg0: tensor<13x21x3xf32>, %arg1: tensor<3xi32>) -> tensor<3x13x21x1xf32> { + // expected-error@+1 {{'tosa.transpose' op expected input tensor rank to equal result tensor rank}} + %0 = tosa.transpose %arg0 {perms = array}: (tensor<13x21x3xf32>) -> tensor<3x13x21x1xf32> + return %0 : tensor<3x13x21x1xf32> +} + +// ----- + +func.func @test_transpose_rank0_perms() { + %14 = tensor.empty() : tensor<5x27xi64> + // expected-error@+1 {{'tosa.transpose' op expected perms attribute to have size 2 (input rank) but got size 0}} + %72 = tosa.transpose %14 {perms = array }: (tensor<5x27xi64>) -> tensor + return +} + +// ----- + +func.func @test_transpose_invalid_perms_size(%arg0: tensor<13x21x3xf32>) -> tensor<3x13x21xf32> { + // expected-error@+1 {{'tosa.transpose' op expected perms attribute to have size 3 (input rank) but got size 7}} + %0 = tosa.transpose %arg0 {perms = array }: (tensor<13x21x3xf32>) -> tensor<3x13x21xf32> + return %0 : tensor<3x13x21xf32> +} + +// ----- + +func.func @test_transpose_invalid_permutation_tensor(%arg0: tensor<13x21x3xf32>) -> tensor { + // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} + %0 = tosa.transpose %arg0 {perms = array }: (tensor<13x21x3xf32>) -> tensor + return %0 : tensor +} + +// ----- + +func.func @test_transpose_invalid_permutation_negative(%arg0: tensor<3x2xi32>) -> tensor<*xi32> { + // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} + %1 = tosa.transpose %arg0 {perms = array }: (tensor<3x2xi32>) -> tensor<*xi32> + return %1 : tensor<*xi32> +} + +// ----- + +func.func @test_transpose_invalid_permutation_tensor_above_range(%arg0: tensor<3x2xi32>) -> tensor<*xi32> { + // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} + %1 = tosa.transpose %arg0 {perms = array }: (tensor<3x2xi32>) -> tensor<*xi32> + return %1 : tensor<*xi32> +} + +// ----- + +func.func @test_transpose_invalid_num_elements(%arg0: tensor<3x2xi32>) -> tensor<3x4xi32> { + // expected-error@+1 {{'tosa.transpose' op expected input1 and output to have same numbers of elements, got 6 and 12}} + %1 = tosa.transpose %arg0 {perms = array }: (tensor<3x2xi32>) -> tensor<3x4xi32> + return %1 : tensor<3x4xi32> +} + +// ----- + +func.func @test_transpose_invalid_permutation_types(%arg0: tensor<3x2xi32>) -> tensor<3x2xi32> { + // expected-error@+1 {{'tosa.transpose' op expected output tensor dim 0 to match input dim 1 with value of 2}} + %1 = tosa.transpose %arg0 {perms = array }: (tensor<3x2xi32>) -> tensor<3x2xi32> + return %1 : tensor<3x2xi32> +} + +// ----- + +func.func @test_transpose_invalid_permutation_types_dynamic_dim_ok(%arg0: tensor<2x?xi32>) -> tensor<3x4xi32> { + // expected-error@+1 {{'tosa.transpose' op expected output tensor dim 1 to match input dim 0 with value of 2}} + %1 = tosa.transpose %arg0 {perms = array }: (tensor<2x?xi32>) -> tensor<3x4xi32> + return %1 : tensor<3x4xi32> +} + +// ----- + +func.func @test_transpose_element_type_mismatch(%arg0: tensor<2x3xi32>) -> tensor<3x2xf32> { + // expected-error@+1 {{'tosa.transpose' op failed to verify that all of {input1, output} have same element type}} + %1 = tosa.transpose %arg0 {perms = array} : (tensor<2x3xi32>) -> tensor<3x2xf32> + return %1 : tensor<3x2xf32> +} + +// ----- + +// CHECK-LABEL: @test_invalid_constant_permutation +func.func @test_invalid_constant_permutation() { + %0 = tensor.empty() : tensor<3x4x5xi32> + // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} + %2 = tosa.transpose %0 {perms = array}: (tensor<3x4x5xi32>) -> tensor<3x4x5xi32> + return +} + +// ----- + +// CHECK-LABEL: test_rank_size_constant_permutation +func.func @test_rank_size_constant_permutation() { + %0 = arith.constant 6 : index + %2 = tensor.empty(%0) : tensor + // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} + %3 = tosa.transpose %2 {perms = array}: (tensor) -> tensor + return +} + +// ----- + +// CHECK-LABEL: test_large_constant_permutation +func.func @test_large_constant_permutation() { + %0 = arith.constant 6 : index + %2 = tensor.empty(%0) : tensor + // expected-error@+1 {{'tosa.transpose' op expected valid permutation indices}} + %3 = tosa.transpose %2 {perms = array}: (tensor) -> tensor + return +} + +// ----- + +func.func @test_scalar_output_transpose(%arg0: tensor<*xf32>) -> tensor { + // expected-error@+1 {{'tosa.transpose' op result #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor'}} + %1 = tosa.transpose %arg0 {perms = array} : (tensor<*xf32>) -> tensor + return %1 : tensor +} From 96064e1b516aba4d7cbea2ab183b20b19b7eea86 Mon Sep 17 00:00:00 2001 From: Tai Ly Date: Tue, 15 Apr 2025 14:36:53 -0500 Subject: [PATCH 16/63] [mlir][tosa] Add table size check for Table Op (#135262) Add table size check for Table Op and add lit tests to error_if_check.mlir also corrected some existing tests that violated the table size checks Signed-off-by: Tai Ly --- .../Tosa/Transforms/TosaValidation.cpp | 24 ++++++++++++++++++- mlir/test/Dialect/Tosa/dynamic_extension.mlir | 4 ++-- mlir/test/Dialect/Tosa/error_if_check.mlir | 16 +++++++++++++ mlir/test/Dialect/Tosa/invalid_extension.mlir | 4 ++-- 4 files changed, 43 insertions(+), 5 deletions(-) diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp index 11eb0d969d78b..ef9d27f8df0ad 100644 --- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp +++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp @@ -1012,8 +1012,30 @@ bool checkErrorIfMul(Operation *op) { return true; } +bool checkErrorIfTable(Operation *op) { + auto table = dyn_cast(op); + if (!table) + return true; + + // REQUIRE(length(table) == TABLE_SIZE) where TABLE_SIZE is 256 or 513 + const auto inputElemType = getElementTypeOrSelf(table.getInput1().getType()); + const int tableSize = inputElemType.isInteger(8) ? 256 : 513; + + const ShapeAdaptor tableShape(table.getTable().getType()); + if (tableShape.hasStaticShape()) { + const auto numElements = tableShape.getNumElements(); + if (numElements != tableSize) { + op->emitOpError() << "requires table size of " << tableSize << ", got " + << numElements; + return false; + } + } + + return true; +} + LogicalResult TosaValidation::applyErrorIfCheck(Operation *op) { - if (!checkErrorIfResize(op) || !checkErrorIfMul(op)) + if (!checkErrorIfResize(op) || !checkErrorIfMul(op) || !checkErrorIfTable(op)) return failure(); return success(); } diff --git a/mlir/test/Dialect/Tosa/dynamic_extension.mlir b/mlir/test/Dialect/Tosa/dynamic_extension.mlir index 0ec46022157d7..25e1aa195c3a0 100644 --- a/mlir/test/Dialect/Tosa/dynamic_extension.mlir +++ b/mlir/test/Dialect/Tosa/dynamic_extension.mlir @@ -13,8 +13,8 @@ func.func @test_mul_non_const(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8 // ----- -func.func @test_table_non_const(%arg0 : tensor<4x5xi8>, %arg1 : tensor<513xi8>) -> () { - %0 = tosa.table %arg0, %arg1 : (tensor<4x5xi8>, tensor<513xi8>) -> tensor<4x5xi8> +func.func @test_table_non_const(%arg0 : tensor<4x5xi8>, %arg1 : tensor<256xi8>) -> () { + %0 = tosa.table %arg0, %arg1 : (tensor<4x5xi8>, tensor<256xi8>) -> tensor<4x5xi8> return } diff --git a/mlir/test/Dialect/Tosa/error_if_check.mlir b/mlir/test/Dialect/Tosa/error_if_check.mlir index f7ca0faa8bc9e..65a69be91e0c8 100644 --- a/mlir/test/Dialect/Tosa/error_if_check.mlir +++ b/mlir/test/Dialect/Tosa/error_if_check.mlir @@ -113,3 +113,19 @@ func.func @test_mul_non_zero_shift(%arg0: tensor<1x8x8x8xi16>, %arg1: tensor<1x8 %mul = tosa.mul %arg0, %arg1, %shift : (tensor<1x8x8x8xi16>, tensor<1x8x8x8xi16>, tensor<1xi8>) -> tensor<1x8x8x8xi32> return %mul : tensor<1x8x8x8xi32> } + +// ----- +// CHECK-LABEL: test_i16_table_size +func.func @test_i16_table_size(%arg0: tensor<2x64xi16>, %arg1: tensor<256xi16>) -> tensor<2x64xi32> { + // expected-error@+1 {{'tosa.table' op requires table size of 513, got 256}} + %0 = tosa.table %arg0, %arg1 : (tensor<2x64xi16>, tensor<256xi16>) -> tensor<2x64xi32> + return %0 : tensor<2x64xi32> +} + +// ----- +// CHECK-LABEL: test_i8_table_size +func.func @test_i8_table_size(%arg0: tensor<2x64xi8>, %arg1: tensor<513xi8>) -> tensor<2x64xi8> { + // expected-error@+1 {{'tosa.table' op requires table size of 256, got 513}} + %0 = tosa.table %arg0, %arg1 : (tensor<2x64xi8>, tensor<513xi8>) -> tensor<2x64xi8> + return %0 : tensor<2x64xi8> +} diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir index 241e603e91c61..7386b1ba9df99 100644 --- a/mlir/test/Dialect/Tosa/invalid_extension.mlir +++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir @@ -497,9 +497,9 @@ func.func @test_mul_non_const(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8 // ----- -func.func @test_table_non_const(%arg0 : tensor<4x5xi8>, %arg1 : tensor<513xi8>) -> () { +func.func @test_table_non_const(%arg0 : tensor<4x5xi8>, %arg1 : tensor<256xi8>) -> () { // expected-error@+1 {{'tosa.table' op expected compile time resolvable constant, but got variable value for operand #1}} - %0 = tosa.table %arg0, %arg1 : (tensor<4x5xi8>, tensor<513xi8>) -> tensor<4x5xi8> + %0 = tosa.table %arg0, %arg1 : (tensor<4x5xi8>, tensor<256xi8>) -> tensor<4x5xi8> return } From 4f64c80d5a23c244f942193e58ecac666c173308 Mon Sep 17 00:00:00 2001 From: Michael Spencer Date: Tue, 15 Apr 2025 12:39:00 -0700 Subject: [PATCH 17/63] [llvm] Add missing include for !LLVM_ENABLE_THREADS thread.h used report_fatal_error without including ErrorHandling.h --- llvm/include/llvm/Support/thread.h | 1 + 1 file changed, 1 insertion(+) diff --git a/llvm/include/llvm/Support/thread.h b/llvm/include/llvm/Support/thread.h index e3005fdb63175..ef2fba822cb1c 100644 --- a/llvm/include/llvm/Support/thread.h +++ b/llvm/include/llvm/Support/thread.h @@ -213,6 +213,7 @@ inline thread::id get_id() { return std::this_thread::get_id(); } #else // !LLVM_ENABLE_THREADS +#include "llvm/Support/ErrorHandling.h" #include namespace llvm { From e6e56f5b6a80c6ce55630d6075475cb363afb149 Mon Sep 17 00:00:00 2001 From: Teresa Johnson Date: Tue, 15 Apr 2025 12:45:18 -0700 Subject: [PATCH 18/63] [MemProf] Handle recursion during stack node update (#135837) If we are replacing a sequence of stack nodes with a single node representing inlined IR, and the stack id sequence contains recursion, we may have already removed some edges. Handle this case correctly by skipping the now removed edge. --- .../Transforms/IPO/MemProfContextDisambiguation.cpp | 7 ++++++- .../MemProfContextDisambiguation/inlined2.ll | 11 +++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/llvm/lib/Transforms/IPO/MemProfContextDisambiguation.cpp b/llvm/lib/Transforms/IPO/MemProfContextDisambiguation.cpp index a5e0251277d8f..561c01cb01f82 100644 --- a/llvm/lib/Transforms/IPO/MemProfContextDisambiguation.cpp +++ b/llvm/lib/Transforms/IPO/MemProfContextDisambiguation.cpp @@ -1711,7 +1711,12 @@ void CallsiteContextGraph:: // edge from the prior node. if (PrevNode) { auto *PrevEdge = CurNode->findEdgeFromCallee(PrevNode); - assert(PrevEdge); + // If the sequence contained recursion, we might have already removed + // some edges during the connectNewNode calls above. + if (!PrevEdge) { + PrevNode = CurNode; + continue; + } set_subtract(PrevEdge->getContextIds(), SavedContextIds); if (PrevEdge->getContextIds().empty()) removeEdgeFromGraph(PrevEdge); diff --git a/llvm/test/Transforms/MemProfContextDisambiguation/inlined2.ll b/llvm/test/Transforms/MemProfContextDisambiguation/inlined2.ll index cc97b5290e25a..2cc655e927d12 100644 --- a/llvm/test/Transforms/MemProfContextDisambiguation/inlined2.ll +++ b/llvm/test/Transforms/MemProfContextDisambiguation/inlined2.ll @@ -40,6 +40,9 @@ ;; in the input IR to ensure that the MIB call chain is matched to the longer ;; inline sequences from main. ;; +;; Update: the inlined sequence of callsite ids was manually modified to +;; include some recursion, which reproduced an error before it was fixed. +;; ;; The IR was then reduced using llvm-reduce with the expected FileCheck input. ; RUN: opt -passes=memprof-context-disambiguation -supports-hot-cold-new \ @@ -96,13 +99,13 @@ attributes #7 = { builtin } !6 = !{i32 7, !"frame-pointer", i32 2} !7 = !{!8, !10} !8 = !{!9, !"notcold"} -!9 = !{i64 9086428284934609951, i64 -5964873800580613432, i64 2732490490862098848, i64 8632435727821051414} +!9 = !{i64 9086428284934609951, i64 -5964873800580613432, i64 2732490490862098848, i64 -5964873800580613432, i64 8632435727821051414} !10 = !{!11, !"cold"} -!11 = !{i64 9086428284934609951, i64 -5964873800580613432, i64 2732490490862098848, i64 -3421689549917153178} +!11 = !{i64 9086428284934609951, i64 -5964873800580613432, i64 2732490490862098848, i64 -5964873800580613432, i64 -3421689549917153178} !12 = !{i64 9086428284934609951} !13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char) -!14 = !{i64 -5964873800580613432, i64 2732490490862098848, i64 8632435727821051414} -!15 = !{i64 -5964873800580613432, i64 2732490490862098848, i64 -3421689549917153178} +!14 = !{i64 -5964873800580613432, i64 2732490490862098848, i64 -5964873800580613432, i64 8632435727821051414} +!15 = !{i64 -5964873800580613432, i64 2732490490862098848, i64 -5964873800580613432, i64 -3421689549917153178} ; DUMP: CCG before cloning: From a3283a92aea147e89d9d404fa7c8500223c7c22a Mon Sep 17 00:00:00 2001 From: Akira Hatanaka Date: Tue, 15 Apr 2025 12:54:25 -0700 Subject: [PATCH 19/63] [PAC] Add support for __ptrauth type qualifier (#100830) The qualifier allows programmer to directly control how pointers are signed when they are stored in a particular variable. The qualifier takes three arguments: the signing key, a flag specifying whether address discrimination should be used, and a non-negative integer that is used for additional discrimination. ``` typedef void (*my_callback)(const void*); my_callback __ptrauth(ptrauth_key_process_dependent_code, 1, 0xe27a) callback; ``` Co-Authored-By: John McCall rjmccall@apple.com --- clang/docs/PointerAuthentication.rst | 46 ++ clang/docs/ReleaseNotes.rst | 2 + clang/include/clang/AST/Type.h | 21 +- clang/include/clang/Basic/Attr.td | 8 + clang/include/clang/Basic/AttrDocs.td | 28 + .../clang/Basic/DiagnosticParseKinds.td | 3 + .../clang/Basic/DiagnosticSemaKinds.td | 43 +- clang/include/clang/Basic/Features.def | 1 + clang/include/clang/Basic/TokenKinds.def | 1 + clang/include/clang/Parse/Parser.h | 2 + clang/include/clang/Sema/Sema.h | 11 + clang/lib/AST/ASTContext.cpp | 1 + clang/lib/AST/DeclCXX.cpp | 27 + clang/lib/AST/ItaniumMangle.cpp | 20 + clang/lib/AST/MicrosoftMangle.cpp | 13 + clang/lib/AST/TypePrinter.cpp | 40 + clang/lib/CodeGen/CGClass.cpp | 3 + clang/lib/CodeGen/CGDebugInfo.cpp | 19 +- clang/lib/CodeGen/CGDecl.cpp | 12 +- clang/lib/CodeGen/CGExpr.cpp | 94 ++- clang/lib/CodeGen/CGExprConstant.cpp | 23 +- clang/lib/CodeGen/CGExprScalar.cpp | 62 ++ clang/lib/CodeGen/CGPointerAuth.cpp | 129 +++ clang/lib/CodeGen/CodeGenFunction.h | 26 +- clang/lib/Parse/ParseDecl.cpp | 52 ++ clang/lib/Sema/SemaCast.cpp | 19 +- clang/lib/Sema/SemaChecking.cpp | 57 ++ clang/lib/Sema/SemaDecl.cpp | 13 +- clang/lib/Sema/SemaDeclCXX.cpp | 38 + clang/lib/Sema/SemaExpr.cpp | 14 + clang/lib/Sema/SemaExprCXX.cpp | 5 + clang/lib/Sema/SemaObjCProperty.cpp | 3 + clang/lib/Sema/SemaOverload.cpp | 15 + clang/lib/Sema/SemaType.cpp | 79 ++ clang/lib/Sema/TreeTransform.h | 11 + clang/test/AST/ast-dump-ptrauth-json.cpp | 3 + clang/test/CodeGen/ptrauth-debuginfo.c | 35 + .../CodeGen/ptrauth-qualifier-const-init.c | 86 ++ .../test/CodeGen/ptrauth-qualifier-function.c | 145 ++++ .../CodeGen/ptrauth-qualifier-loadstore.c | 745 ++++++++++++++++++ .../CodeGenCXX/mangle-itanium-ptrauth.cpp | 12 + clang/test/CodeGenCXX/mangle-ms-ptrauth.cpp | 17 + .../CodeGenCXX/ptrauth-qualifier-struct.cpp | 168 ++++ .../CodeGenObjCXX/ptrauth-struct-cxx-abi.mm | 35 + clang/test/Parser/ptrauth-qualifier.c | 18 + clang/test/Preprocessor/ptrauth_extension.c | 13 + clang/test/Sema/ptrauth-atomic-ops.c | 118 +++ clang/test/Sema/ptrauth-qualifier.c | 103 +++ clang/test/SemaCXX/ptrauth-qualifier.cpp | 213 +++++ .../SemaCXX/ptrauth-template-parameters.cpp | 29 + clang/test/SemaObjC/ptrauth-qualifier.m | 56 ++ libcxxabi/test/test_demangle.pass.cpp | 3 + .../include/llvm/Demangle/MicrosoftDemangle.h | 8 + .../llvm/Demangle/MicrosoftDemangleNodes.h | 22 +- llvm/lib/Demangle/MicrosoftDemangle.cpp | 48 +- llvm/lib/Demangle/MicrosoftDemangleNodes.cpp | 10 + llvm/test/Demangle/ms-ptrauth.test | 12 + 57 files changed, 2801 insertions(+), 39 deletions(-) create mode 100644 clang/test/CodeGen/ptrauth-debuginfo.c create mode 100644 clang/test/CodeGen/ptrauth-qualifier-const-init.c create mode 100644 clang/test/CodeGen/ptrauth-qualifier-function.c create mode 100644 clang/test/CodeGen/ptrauth-qualifier-loadstore.c create mode 100644 clang/test/CodeGenCXX/mangle-itanium-ptrauth.cpp create mode 100644 clang/test/CodeGenCXX/mangle-ms-ptrauth.cpp create mode 100644 clang/test/CodeGenCXX/ptrauth-qualifier-struct.cpp create mode 100644 clang/test/CodeGenObjCXX/ptrauth-struct-cxx-abi.mm create mode 100644 clang/test/Parser/ptrauth-qualifier.c create mode 100644 clang/test/Preprocessor/ptrauth_extension.c create mode 100644 clang/test/Sema/ptrauth-atomic-ops.c create mode 100644 clang/test/Sema/ptrauth-qualifier.c create mode 100644 clang/test/SemaCXX/ptrauth-qualifier.cpp create mode 100644 clang/test/SemaCXX/ptrauth-template-parameters.cpp create mode 100644 clang/test/SemaObjC/ptrauth-qualifier.m create mode 100644 llvm/test/Demangle/ms-ptrauth.test diff --git a/clang/docs/PointerAuthentication.rst b/clang/docs/PointerAuthentication.rst index 68674f318c84f..41818d43ac687 100644 --- a/clang/docs/PointerAuthentication.rst +++ b/clang/docs/PointerAuthentication.rst @@ -280,6 +280,52 @@ a number of different tests. normal interface. This may be true even on targets where pointer authentication is not enabled by default. +__ptrauth Qualifier +^^^^^^^^^^^^^^^^^^^ + +``__ptrauth(key, address, discriminator)`` is an extended type +qualifier which causes so-qualified objects to hold pointers signed using the +specified schema rather than the default schema for such types. + +In the current implementation in Clang, the qualified type must be a C pointer +type, either to a function or to an object. It currently cannot be an +Objective-C pointer type, a C++ reference type, or a block pointer type; these +restrictions may be lifted in the future. + +The qualifier's operands are as follows: + +- ``key`` - an expression evaluating to a key value from ````; must + be a constant expression + +- ``address`` - whether to use address diversity (1) or not (0); must be + a constant expression with one of these two values + +- ``discriminator`` - a constant discriminator; must be a constant expression + +See `Discriminators`_ for more information about discriminators. + +Currently the operands must be constant-evaluable even within templates. In the +future this restriction may be lifted to allow value-dependent expressions as +long as they instantiate to a constant expression. + +Consistent with the ordinary C/C++ rule for parameters, top-level ``__ptrauth`` +qualifiers on a parameter (after parameter type adjustment) are ignored when +deriving the type of the function. The parameter will be passed using the +default ABI for the unqualified pointer type. + +If ``x`` is an object of type ``__ptrauth(key, address, discriminator) T``, +then the signing schema of the value stored in ``x`` is a key of ``key`` and +a discriminator determined as follows: + +- if ``address`` is 0, then the discriminator is ``discriminator``; + +- if ``address`` is 1 and ``discriminator`` is 0, then the discriminator is + ``&x``; otherwise + +- if ``address`` is 1 and ``discriminator`` is non-zero, then the discriminator + is ``ptrauth_blend_discriminator(&x, discriminator)``; see + `ptrauth_blend_discriminator`_. + ```` ~~~~~~~~~~~~~~~ diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst index 6025e76029d19..38142ad32bea0 100644 --- a/clang/docs/ReleaseNotes.rst +++ b/clang/docs/ReleaseNotes.rst @@ -552,6 +552,8 @@ Arm and AArch64 Support ARM targets, however this will now disable NEON instructions being generated. The ``simd`` option is also now printed when the ``--print-supported-extensions`` option is used. +- Support for __ptrauth type qualifier has been added. + Android Support ^^^^^^^^^^^^^^^ diff --git a/clang/include/clang/AST/Type.h b/clang/include/clang/AST/Type.h index 74886ef0cd824..5bf036e3347eb 100644 --- a/clang/include/clang/AST/Type.h +++ b/clang/include/clang/AST/Type.h @@ -312,6 +312,12 @@ class PointerAuthQualifier { return Result; } + std::string getAsString() const; + std::string getAsString(const PrintingPolicy &Policy) const; + + bool isEmptyWhenPrinted(const PrintingPolicy &Policy) const; + void print(raw_ostream &OS, const PrintingPolicy &Policy) const; + void Profile(llvm::FoldingSetNodeID &ID) const { ID.AddInteger(Data); } }; @@ -562,7 +568,7 @@ class Qualifiers { bool hasAddressSpace() const { return Mask & AddressSpaceMask; } LangAS getAddressSpace() const { - return static_cast(Mask >> AddressSpaceShift); + return static_cast((Mask & AddressSpaceMask) >> AddressSpaceShift); } bool hasTargetSpecificAddressSpace() const { return isTargetAddressSpace(getAddressSpace()); @@ -803,6 +809,9 @@ class Qualifiers { static_assert(sizeof(PointerAuthQualifier) == sizeof(uint32_t), "PointerAuthQualifier must be 32 bits"); + static constexpr uint64_t PtrAuthShift = 32; + static constexpr uint64_t PtrAuthMask = UINT64_C(0xffffffff) << PtrAuthShift; + static constexpr uint64_t UMask = 0x8; static constexpr uint64_t UShift = 3; static constexpr uint64_t GCAttrMask = 0x30; @@ -810,10 +819,8 @@ class Qualifiers { static constexpr uint64_t LifetimeMask = 0x1C0; static constexpr uint64_t LifetimeShift = 6; static constexpr uint64_t AddressSpaceMask = - ~(CVRMask | UMask | GCAttrMask | LifetimeMask); + ~(CVRMask | UMask | GCAttrMask | LifetimeMask | PtrAuthMask); static constexpr uint64_t AddressSpaceShift = 9; - static constexpr uint64_t PtrAuthShift = 32; - static constexpr uint64_t PtrAuthMask = uint64_t(0xffffffff) << PtrAuthShift; }; class QualifiersAndAtomic { @@ -1449,6 +1456,12 @@ class QualType { return getQualifiers().getPointerAuth(); } + bool hasAddressDiscriminatedPointerAuth() const { + if (PointerAuthQualifier PtrAuth = getPointerAuth()) + return PtrAuth.isAddressDiscriminated(); + return false; + } + enum PrimitiveDefaultInitializeKind { /// The type does not fall into any of the following categories. Note that /// this case is zero-valued so that values of this enum can be used as a diff --git a/clang/include/clang/Basic/Attr.td b/clang/include/clang/Basic/Attr.td index b7ad432738b29..9d4900f3029c8 100644 --- a/clang/include/clang/Basic/Attr.td +++ b/clang/include/clang/Basic/Attr.td @@ -3548,6 +3548,14 @@ def ObjCRequiresPropertyDefs : InheritableAttr { let SimpleHandler = 1; } +def PointerAuth : TypeAttr { + let Spellings = [CustomKeyword<"__ptrauth">]; + let Args = [IntArgument<"Key">, + BoolArgument<"AddressDiscriminated", 1>, + IntArgument<"ExtraDiscriminator", 1>]; + let Documentation = [PtrAuthDocs]; +} + def Unused : InheritableAttr { let Spellings = [CXX11<"", "maybe_unused", 201603>, GCC<"unused">, C23<"", "maybe_unused", 202106>]; diff --git a/clang/include/clang/Basic/AttrDocs.td b/clang/include/clang/Basic/AttrDocs.td index 97a5f24d35d7d..76f805ef373dd 100644 --- a/clang/include/clang/Basic/AttrDocs.td +++ b/clang/include/clang/Basic/AttrDocs.td @@ -2179,6 +2179,34 @@ Also see the documentation for `@available }]; } +def PtrAuthDocs : Documentation { + let Category = DocCatVariable; + let Heading = "__ptrauth"; + let Content = [{ +The ``__ptrauth`` qualifier allows the programmer to directly control +how pointers are signed when they are stored in a particular variable. +This can be used to strengthen the default protections of pointer +authentication and make it more difficult for an attacker to escalate +an ability to alter memory into full control of a process. + +.. code-block:: c + + #include + + typedef void (*my_callback)(const void*); + my_callback __ptrauth(ptrauth_key_process_dependent_code, 1, 0xe27a) callback; + +The first argument to ``__ptrauth`` is the name of the signing key. +Valid key names for the target are defined in ````. + +The second argument to ``__ptrauth`` is a flag (0 or 1) specifying whether +the object should use address discrimination. + +The third argument to ``__ptrauth`` is a 16-bit non-negative integer which +allows additional discrimination between objects. + }]; +} + def ExternalSourceSymbolDocs : Documentation { let Category = DocCatDecl; let Content = [{ diff --git a/clang/include/clang/Basic/DiagnosticParseKinds.td b/clang/include/clang/Basic/DiagnosticParseKinds.td index 7a3cac528a363..9975520f4f9ff 100644 --- a/clang/include/clang/Basic/DiagnosticParseKinds.td +++ b/clang/include/clang/Basic/DiagnosticParseKinds.td @@ -1721,6 +1721,9 @@ def warn_pragma_unroll_cuda_value_in_parens : Warning< "argument to '#pragma unroll' should not be in parentheses in CUDA C/C++">, InGroup; +def err_ptrauth_qualifier_bad_arg_count : Error< + "'__ptrauth' qualifier must take between 1 and 3 arguments">; + def warn_cuda_attr_lambda_position : Warning< "nvcc does not allow '__%0__' to appear after the parameter list in lambdas">, InGroup; diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td index f4ab620ae61d2..3f7499d8656bd 100644 --- a/clang/include/clang/Basic/DiagnosticSemaKinds.td +++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td @@ -1014,6 +1014,22 @@ def err_ptrauth_indirect_goto_addrlabel_arithmetic : Error< "%select{subtraction|addition}0 of address-of-label expressions is not " "supported with ptrauth indirect gotos">; +// __ptrauth qualifier +def err_ptrauth_qualifier_invalid : Error< + "%select{return type|parameter type|property}1 may not be qualified with '__ptrauth'; type is %0">; +def err_ptrauth_qualifier_cast : Error< + "cannot cast to '__ptrauth'-qualified type %0">; +def err_ptrauth_qualifier_nonpointer : Error< + "'__ptrauth' qualifier only applies to pointer types; %0 is invalid">; +def err_ptrauth_qualifier_redundant : Error< + "type %0 is already %1-qualified">; +def err_ptrauth_arg_not_ice : Error< + "argument to '__ptrauth' must be an integer constant expression">; +def err_ptrauth_address_discrimination_invalid : Error< + "invalid address discrimination flag '%0'; '__ptrauth' requires '0' or '1'">; +def err_ptrauth_extra_discriminator_invalid : Error< + "invalid extra discriminator flag '%0'; '__ptrauth' requires a value between '0' and '%1'">; + /// main() // static main() is not an error in C, just in C++. def warn_static_main : Warning<"'main' should not be declared static">, @@ -3923,7 +3939,8 @@ def note_cannot_use_trivial_abi_reason : Note< "its copy constructors and move constructors are all deleted|" "it is polymorphic|" "it has a base of a non-trivial class type|it has a virtual base|" - "it has a __weak field|it has a field of a non-trivial class type}1">; + "it has a __weak field|it has a field of a non-trivial class type|" + "it has an address-discriminated '__ptrauth' field}1">; // Availability attribute def warn_availability_unknown_platform : Warning< @@ -5021,6 +5038,10 @@ def note_ovl_candidate_bad_ownership : Note< "%select{no|__unsafe_unretained|__strong|__weak|__autoreleasing}4 ownership," " but parameter has %select{no|__unsafe_unretained|__strong|__weak|" "__autoreleasing}5 ownership">; +def note_ovl_candidate_bad_ptrauth : Note< + "candidate %sub{select_ovl_candidate_kind}0,1,2 not viable: " + "%ordinal8 argument (%3) has %select{no '__ptrauth'|%5}4 qualifier," + " but parameter has %select{no '__ptrauth'|%7}6 qualifier">; def note_ovl_candidate_bad_cvr_this : Note< "candidate %sub{select_ovl_candidate_kind}0,1,2 not viable: " "'this' argument has type %3, but method is not marked " @@ -6092,7 +6113,7 @@ def note_deleted_special_member_class_subobject : Note< "%select{default|corresponding|default|default|default}4 constructor}0|" "destructor}5" "%select{||s||}4" - "|is an ObjC pointer}6">; + "|is an ObjC pointer|has an address-discriminated '__ptrauth' qualifier}6">; def note_default_constructed_field : Note<"default constructed field %0 declared here">; def note_deleted_default_ctor_uninit_field : Note< @@ -8938,6 +8959,19 @@ def err_typecheck_incompatible_ownership : Error< "sending to parameter of different type}0,1" "|%diff{casting $ to type $|casting between types}0,1}2" " changes retain/release properties of pointer">; +def err_typecheck_incompatible_ptrauth : Error< + "%enum_select{%Assigning{%diff{assigning $ to $|assigning to different types}1,0}" + "|%Passing{%diff{passing $ to parameter of type $|" + "passing to parameter of different type}0,1}" + "|%Returning{%diff{returning $ from a function with result type $|" + "returning from function with different return type}0,1}" + "|%Converting{%diff{converting $ to type $|converting between types}0,1}" + "|%Initializing{%diff{initializing $ with an expression of type $|" + "initializing with expression of different type}0,1}" + "|%Sending{%diff{sending $ to parameter of type $|" + "sending to parameter of different type}0,1}" + "|%Casting{%diff{casting $ to type $|casting between types}0,1}}2" + " changes pointer authentication of pointee type">; def err_typecheck_comparison_of_distinct_blocks : Error< "comparison of distinct block types%diff{ ($ and $)|}0,1">; @@ -9066,6 +9100,9 @@ def err_atomic_op_needs_non_const_atomic : Error< def err_atomic_op_needs_non_const_pointer : Error< "address argument to atomic operation must be a pointer to non-const " "type (%0 invalid)">; +def err_atomic_op_needs_non_address_discriminated_pointer : Error< + "address argument to %select{atomic|__sync}0 operation must be a pointer to a non address discriminated " + "type (%1 invalid)">; def err_atomic_op_needs_trivial_copy : Error< "address argument to atomic operation must be a pointer to a " "trivially-copyable type (%0 invalid)">; @@ -9343,6 +9380,8 @@ def ext_typecheck_cond_pointer_integer_mismatch : ExtWarn< "pointer/integer type mismatch in conditional expression" "%diff{ ($ and $)|}0,1">, InGroup>; +def err_typecheck_cond_incompatible_ptrauth : Error< + "'__ptrauth' qualification mismatch%diff{ ($ and $)|}0,1">; def err_typecheck_choose_expr_requires_constant : Error< "'__builtin_choose_expr' requires a constant expression">; def warn_unused_expr : Warning<"expression result unused">, diff --git a/clang/include/clang/Basic/Features.def b/clang/include/clang/Basic/Features.def index b4409efaa9c04..14bff8a68846d 100644 --- a/clang/include/clang/Basic/Features.def +++ b/clang/include/clang/Basic/Features.def @@ -107,6 +107,7 @@ FEATURE(thread_sanitizer, LangOpts.Sanitize.has(SanitizerKind::Thread)) FEATURE(dataflow_sanitizer, LangOpts.Sanitize.has(SanitizerKind::DataFlow)) FEATURE(scudo, LangOpts.Sanitize.hasOneOf(SanitizerKind::Scudo)) FEATURE(ptrauth_intrinsics, LangOpts.PointerAuthIntrinsics) +EXTENSION(ptrauth_qualifier, LangOpts.PointerAuthIntrinsics) FEATURE(ptrauth_calls, LangOpts.PointerAuthCalls) FEATURE(ptrauth_returns, LangOpts.PointerAuthReturns) FEATURE(ptrauth_vtable_pointer_address_discrimination, LangOpts.PointerAuthVTPtrAddressDiscrimination) diff --git a/clang/include/clang/Basic/TokenKinds.def b/clang/include/clang/Basic/TokenKinds.def index 880928ae0447d..868e851342eb8 100644 --- a/clang/include/clang/Basic/TokenKinds.def +++ b/clang/include/clang/Basic/TokenKinds.def @@ -348,6 +348,7 @@ KEYWORD(_Thread_local , KEYALL) KEYWORD(__func__ , KEYALL) KEYWORD(__objc_yes , KEYALL) KEYWORD(__objc_no , KEYALL) +KEYWORD(__ptrauth , KEYALL) // C2y UNARY_EXPR_OR_TYPE_TRAIT(_Countof, CountOf, KEYNOCXX) diff --git a/clang/include/clang/Parse/Parser.h b/clang/include/clang/Parse/Parser.h index 53da6269a3b11..9ebcf144ba59e 100644 --- a/clang/include/clang/Parse/Parser.h +++ b/clang/include/clang/Parse/Parser.h @@ -3169,6 +3169,8 @@ class Parser : public CodeCompletionHandler { SourceLocation *endLoc = nullptr); ExprResult ParseExtIntegerArgument(); + void ParsePtrauthQualifier(ParsedAttributes &Attrs); + VirtSpecifiers::Specifier isCXX11VirtSpecifier(const Token &Tok) const; VirtSpecifiers::Specifier isCXX11VirtSpecifier() const { return isCXX11VirtSpecifier(Tok); diff --git a/clang/include/clang/Sema/Sema.h b/clang/include/clang/Sema/Sema.h index 5ab0af8234e26..fe37fd7701ce3 100644 --- a/clang/include/clang/Sema/Sema.h +++ b/clang/include/clang/Sema/Sema.h @@ -3547,6 +3547,17 @@ class Sema final : public SemaBase { bool checkConstantPointerAuthKey(Expr *keyExpr, unsigned &key); + enum PointerAuthDiscArgKind { + // Address discrimination argument of __ptrauth. + PADAK_AddrDiscPtrAuth, + + // Extra discriminator argument of __ptrauth. + PADAK_ExtraDiscPtrAuth, + }; + + bool checkPointerAuthDiscriminatorArg(Expr *Arg, PointerAuthDiscArgKind Kind, + unsigned &IntVal); + /// Diagnose function specifiers on a declaration of an identifier that /// does not identify a function. void DiagnoseFunctionSpecifiers(const DeclSpec &DS); diff --git a/clang/lib/AST/ASTContext.cpp b/clang/lib/AST/ASTContext.cpp index dfae2f5511b43..c6ffe7bbf5257 100644 --- a/clang/lib/AST/ASTContext.cpp +++ b/clang/lib/AST/ASTContext.cpp @@ -11454,6 +11454,7 @@ QualType ASTContext::mergeTypes(QualType LHS, QualType RHS, bool OfBlockPointer, if (LQuals.getCVRQualifiers() != RQuals.getCVRQualifiers() || LQuals.getAddressSpace() != RQuals.getAddressSpace() || LQuals.getObjCLifetime() != RQuals.getObjCLifetime() || + !LQuals.getPointerAuth().isEquivalent(RQuals.getPointerAuth()) || LQuals.hasUnaligned() != RQuals.hasUnaligned()) return {}; diff --git a/clang/lib/AST/DeclCXX.cpp b/clang/lib/AST/DeclCXX.cpp index 36131d19cbcdf..4d07efd58f518 100644 --- a/clang/lib/AST/DeclCXX.cpp +++ b/clang/lib/AST/DeclCXX.cpp @@ -1118,6 +1118,33 @@ void CXXRecordDecl::addedMember(Decl *D) { } else if (!T.isCXX98PODType(Context)) data().PlainOldData = false; + // If a class has an address-discriminated signed pointer member, it is a + // non-POD type and its copy constructor, move constructor, copy assignment + // operator, move assignment operator are non-trivial. + if (PointerAuthQualifier Q = T.getPointerAuth()) { + if (Q.isAddressDiscriminated()) { + struct DefinitionData &Data = data(); + Data.PlainOldData = false; + Data.HasTrivialSpecialMembers &= + ~(SMF_CopyConstructor | SMF_MoveConstructor | SMF_CopyAssignment | + SMF_MoveAssignment); + setArgPassingRestrictions(RecordArgPassingKind::CanNeverPassInRegs); + + // Copy/move constructors/assignment operators of a union are deleted by + // default if it has an address-discriminated ptrauth field. + if (isUnion()) { + data().DefaultedCopyConstructorIsDeleted = true; + data().DefaultedMoveConstructorIsDeleted = true; + data().DefaultedCopyAssignmentIsDeleted = true; + data().DefaultedMoveAssignmentIsDeleted = true; + data().NeedOverloadResolutionForCopyConstructor = true; + data().NeedOverloadResolutionForMoveConstructor = true; + data().NeedOverloadResolutionForCopyAssignment = true; + data().NeedOverloadResolutionForMoveAssignment = true; + } + } + } + if (Field->hasAttr()) setHasUninitializedExplicitInitFields(true); diff --git a/clang/lib/AST/ItaniumMangle.cpp b/clang/lib/AST/ItaniumMangle.cpp index 140f29b431fc3..d0ab60700cb15 100644 --- a/clang/lib/AST/ItaniumMangle.cpp +++ b/clang/lib/AST/ItaniumMangle.cpp @@ -2895,6 +2895,26 @@ void CXXNameMangler::mangleQualifiers(Qualifiers Quals, const DependentAddressSp if (Quals.hasUnaligned()) mangleVendorQualifier("__unaligned"); + // __ptrauth. Note that this is parameterized. + if (PointerAuthQualifier PtrAuth = Quals.getPointerAuth()) { + mangleVendorQualifier("__ptrauth"); + // For now, since we only allow non-dependent arguments, we can just + // inline the mangling of those arguments as literals. We treat the + // key and extra-discriminator arguments as 'unsigned int' and the + // address-discriminated argument as 'bool'. + Out << "I" + "Lj" + << PtrAuth.getKey() + << "E" + "Lb" + << unsigned(PtrAuth.isAddressDiscriminated()) + << "E" + "Lj" + << PtrAuth.getExtraDiscriminator() + << "E" + "E"; + } + // Remaining ARC ownership qualifiers. switch (Quals.getObjCLifetime()) { case Qualifiers::OCL_None: diff --git a/clang/lib/AST/MicrosoftMangle.cpp b/clang/lib/AST/MicrosoftMangle.cpp index a6efd887d4e13..20bfb7f89625b 100644 --- a/clang/lib/AST/MicrosoftMangle.cpp +++ b/clang/lib/AST/MicrosoftMangle.cpp @@ -430,6 +430,7 @@ class MicrosoftCXXNameMangler { void mangleRefQualifier(RefQualifierKind RefQualifier); void manglePointerCVQualifiers(Qualifiers Quals); void manglePointerExtQualifiers(Qualifiers Quals, QualType PointeeType); + void manglePointerAuthQualifier(Qualifiers Quals); void mangleUnscopedTemplateName(GlobalDecl GD); void @@ -2340,6 +2341,17 @@ void MicrosoftCXXNameMangler::manglePointerExtQualifiers(Qualifiers Quals, Out << 'F'; } +void MicrosoftCXXNameMangler::manglePointerAuthQualifier(Qualifiers Quals) { + PointerAuthQualifier PointerAuth = Quals.getPointerAuth(); + if (!PointerAuth) + return; + + Out << "__ptrauth"; + mangleNumber(PointerAuth.getKey()); + mangleNumber(PointerAuth.isAddressDiscriminated()); + mangleNumber(PointerAuth.getExtraDiscriminator()); +} + void MicrosoftCXXNameMangler::manglePointerCVQualifiers(Qualifiers Quals) { // ::= P # no qualifiers // ::= Q # const @@ -3372,6 +3384,7 @@ void MicrosoftCXXNameMangler::mangleType(const PointerType *T, Qualifiers Quals, QualType PointeeType = T->getPointeeType(); manglePointerCVQualifiers(Quals); manglePointerExtQualifiers(Quals, PointeeType); + manglePointerAuthQualifier(Quals); // For pointer size address spaces, go down the same type mangling path as // non address space types. diff --git a/clang/lib/AST/TypePrinter.cpp b/clang/lib/AST/TypePrinter.cpp index cc8c874140167..7b1d1c7ae2131 100644 --- a/clang/lib/AST/TypePrinter.cpp +++ b/clang/lib/AST/TypePrinter.cpp @@ -2016,6 +2016,7 @@ void TypePrinter::printAttributedAfter(const AttributedType *T, case attr::Ptr64: case attr::SPtr: case attr::UPtr: + case attr::PointerAuth: case attr::AddressSpace: case attr::CmseNSCall: case attr::AnnotateType: @@ -2512,6 +2513,33 @@ void clang::printTemplateArgumentList(raw_ostream &OS, printTo(OS, Args, InnerPolicy, TPL, /*isPack*/ false, /*parmIndex*/ 0); } +std::string PointerAuthQualifier::getAsString() const { + LangOptions LO; + return getAsString(PrintingPolicy(LO)); +} + +std::string PointerAuthQualifier::getAsString(const PrintingPolicy &P) const { + SmallString<64> Buf; + llvm::raw_svector_ostream StrOS(Buf); + print(StrOS, P); + return StrOS.str().str(); +} + +bool PointerAuthQualifier::isEmptyWhenPrinted(const PrintingPolicy &P) const { + return !isPresent(); +} + +void PointerAuthQualifier::print(raw_ostream &OS, + const PrintingPolicy &P) const { + if (!isPresent()) + return; + + OS << "__ptrauth("; + OS << getKey(); + OS << "," << unsigned(isAddressDiscriminated()) << "," + << getExtraDiscriminator() << ")"; +} + std::string Qualifiers::getAsString() const { LangOptions LO; return getAsString(PrintingPolicy(LO)); @@ -2541,6 +2569,10 @@ bool Qualifiers::isEmptyWhenPrinted(const PrintingPolicy &Policy) const { if (!(lifetime == Qualifiers::OCL_Strong && Policy.SuppressStrongLifetime)) return false; + if (PointerAuthQualifier PointerAuth = getPointerAuth(); + PointerAuth && !PointerAuth.isEmptyWhenPrinted(Policy)) + return false; + return true; } @@ -2651,6 +2683,14 @@ void Qualifiers::print(raw_ostream &OS, const PrintingPolicy& Policy, } } + if (PointerAuthQualifier PointerAuth = getPointerAuth()) { + if (addSpace) + OS << ' '; + addSpace = true; + + PointerAuth.print(OS, Policy); + } + if (appendSpaceIfNonEmpty && addSpace) OS << ' '; } diff --git a/clang/lib/CodeGen/CGClass.cpp b/clang/lib/CodeGen/CGClass.cpp index 1c73a5bf75f37..7176fe025b386 100644 --- a/clang/lib/CodeGen/CGClass.cpp +++ b/clang/lib/CodeGen/CGClass.cpp @@ -925,6 +925,9 @@ namespace { Qualifiers Qual = F->getType().getQualifiers(); if (Qual.hasVolatile() || Qual.hasObjCLifetime()) return false; + if (PointerAuthQualifier Q = F->getType().getPointerAuth(); + Q && Q.isAddressDiscriminated()) + return false; return true; } diff --git a/clang/lib/CodeGen/CGDebugInfo.cpp b/clang/lib/CodeGen/CGDebugInfo.cpp index e63341c180420..f3ec498d4064b 100644 --- a/clang/lib/CodeGen/CGDebugInfo.cpp +++ b/clang/lib/CodeGen/CGDebugInfo.cpp @@ -1059,8 +1059,23 @@ llvm::DIType *CGDebugInfo::CreateQualifiedType(QualType Ty, // additional ones. llvm::dwarf::Tag Tag = getNextQualifier(Qc); if (!Tag) { - assert(Qc.empty() && "Unknown type qualifier for debug info"); - return getOrCreateType(QualType(T, 0), Unit); + if (Qc.getPointerAuth()) { + unsigned Key = Qc.getPointerAuth().getKey(); + bool IsDiscr = Qc.getPointerAuth().isAddressDiscriminated(); + unsigned ExtraDiscr = Qc.getPointerAuth().getExtraDiscriminator(); + bool IsaPointer = Qc.getPointerAuth().isIsaPointer(); + bool AuthenticatesNullValues = + Qc.getPointerAuth().authenticatesNullValues(); + Qc.removePointerAuth(); + assert(Qc.empty() && "Unknown type qualifier for debug info"); + llvm::DIType *FromTy = getOrCreateType(QualType(T, 0), Unit); + return DBuilder.createPtrAuthQualifiedType(FromTy, Key, IsDiscr, + ExtraDiscr, IsaPointer, + AuthenticatesNullValues); + } else { + assert(Qc.empty() && "Unknown type qualifier for debug info"); + return getOrCreateType(QualType(T, 0), Unit); + } } auto *FromTy = getOrCreateType(Qc.apply(CGM.getContext(), T), Unit); diff --git a/clang/lib/CodeGen/CGDecl.cpp b/clang/lib/CodeGen/CGDecl.cpp index db8dbf86eca4f..db34e2738b4cf 100644 --- a/clang/lib/CodeGen/CGDecl.cpp +++ b/clang/lib/CodeGen/CGDecl.cpp @@ -776,11 +776,17 @@ void CodeGenFunction::EmitScalarInit(const Expr *init, const ValueDecl *D, LValue lvalue, bool capturedByInit) { Qualifiers::ObjCLifetime lifetime = lvalue.getObjCLifetime(); if (!lifetime) { - llvm::Value *value = EmitScalarExpr(init); + llvm::Value *Value; + if (PointerAuthQualifier PtrAuth = lvalue.getQuals().getPointerAuth()) { + Value = EmitPointerAuthQualify(PtrAuth, init, lvalue.getAddress()); + lvalue.getQuals().removePointerAuth(); + } else { + Value = EmitScalarExpr(init); + } if (capturedByInit) drillIntoBlockVariable(*this, lvalue, cast(D)); - EmitNullabilityCheck(lvalue, value, init->getExprLoc()); - EmitStoreThroughLValue(RValue::get(value), lvalue, true); + EmitNullabilityCheck(lvalue, Value, init->getExprLoc()); + EmitStoreThroughLValue(RValue::get(Value), lvalue, true); return; } diff --git a/clang/lib/CodeGen/CGExpr.cpp b/clang/lib/CodeGen/CGExpr.cpp index 3da21cebd9d68..abb88477062fc 100644 --- a/clang/lib/CodeGen/CGExpr.cpp +++ b/clang/lib/CodeGen/CGExpr.cpp @@ -569,7 +569,15 @@ EmitMaterializeTemporaryExpr(const MaterializeTemporaryExpr *M) { // initialized it. if (!Var->hasInitializer()) { Var->setInitializer(CGM.EmitNullConstant(E->getType())); - EmitAnyExprToMem(E, Object, Qualifiers(), /*IsInit*/true); + QualType RefType = M->getType().withoutLocalFastQualifiers(); + if (RefType.getPointerAuth()) { + // Use the qualifier of the reference temporary to sign the pointer. + LValue LV = MakeRawAddrLValue(Object.getPointer(), RefType, + Object.getAlignment()); + EmitScalarInit(E, M->getExtendingDecl(), LV, false); + } else { + EmitAnyExprToMem(E, Object, Qualifiers(), /*IsInit*/ true); + } } } else { switch (M->getStorageDuration()) { @@ -1770,16 +1778,16 @@ static ConstantEmissionKind checkVarTypeForConstantEmission(QualType type) { /// for instance if a block or lambda or a member of a local class uses a /// const int variable or constexpr variable from an enclosing function. CodeGenFunction::ConstantEmission -CodeGenFunction::tryEmitAsConstant(DeclRefExpr *refExpr) { - ValueDecl *value = refExpr->getDecl(); +CodeGenFunction::tryEmitAsConstant(const DeclRefExpr *RefExpr) { + const ValueDecl *Value = RefExpr->getDecl(); // The value needs to be an enum constant or a constant variable. ConstantEmissionKind CEK; - if (isa(value)) { + if (isa(Value)) { CEK = CEK_None; - } else if (auto *var = dyn_cast(value)) { + } else if (const auto *var = dyn_cast(Value)) { CEK = checkVarTypeForConstantEmission(var->getType()); - } else if (isa(value)) { + } else if (isa(Value)) { CEK = CEK_AsValueOnly; } else { CEK = CEK_None; @@ -1792,15 +1800,15 @@ CodeGenFunction::tryEmitAsConstant(DeclRefExpr *refExpr) { // It's best to evaluate all the way as an r-value if that's permitted. if (CEK != CEK_AsReferenceOnly && - refExpr->EvaluateAsRValue(result, getContext())) { + RefExpr->EvaluateAsRValue(result, getContext())) { resultIsReference = false; - resultType = refExpr->getType(); + resultType = RefExpr->getType().getUnqualifiedType(); // Otherwise, try to evaluate as an l-value. } else if (CEK != CEK_AsValueOnly && - refExpr->EvaluateAsLValue(result, getContext())) { + RefExpr->EvaluateAsLValue(result, getContext())) { resultIsReference = true; - resultType = value->getType(); + resultType = Value->getType(); // Failure. } else { @@ -1819,7 +1827,7 @@ CodeGenFunction::tryEmitAsConstant(DeclRefExpr *refExpr) { // accessible on device. The DRE of the captured reference variable has to be // loaded from captures. if (CGM.getLangOpts().CUDAIsDevice && result.Val.isLValue() && - refExpr->refersToEnclosingVariableOrCapture()) { + RefExpr->refersToEnclosingVariableOrCapture()) { auto *MD = dyn_cast_or_null(CurCodeDecl); if (isLambdaMethod(MD) && MD->getOverloadedOperator() == OO_Call) { const APValue::LValueBase &base = result.Val.getLValueBase(); @@ -1834,17 +1842,17 @@ CodeGenFunction::tryEmitAsConstant(DeclRefExpr *refExpr) { } // Emit as a constant. - auto C = ConstantEmitter(*this).emitAbstract(refExpr->getLocation(), - result.Val, resultType); + llvm::Constant *C = ConstantEmitter(*this).emitAbstract( + RefExpr->getLocation(), result.Val, resultType); // Make sure we emit a debug reference to the global variable. // This should probably fire even for - if (isa(value)) { - if (!getContext().DeclMustBeEmitted(cast(value))) - EmitDeclRefExprDbgValue(refExpr, result.Val); + if (isa(Value)) { + if (!getContext().DeclMustBeEmitted(cast(Value))) + EmitDeclRefExprDbgValue(RefExpr, result.Val); } else { - assert(isa(value)); - EmitDeclRefExprDbgValue(refExpr, result.Val); + assert(isa(Value)); + EmitDeclRefExprDbgValue(RefExpr, result.Val); } // If we emitted a reference constant, we need to dereference that. @@ -2235,6 +2243,15 @@ RValue CodeGenFunction::EmitLoadOfAnyValue(LValue LV, AggValueSlot Slot, /// method emits the address of the lvalue, then loads the result as an rvalue, /// returning the rvalue. RValue CodeGenFunction::EmitLoadOfLValue(LValue LV, SourceLocation Loc) { + // Load from __ptrauth. + if (PointerAuthQualifier PtrAuth = LV.getQuals().getPointerAuth()) { + LV.getQuals().removePointerAuth(); + llvm::Value *Value = EmitLoadOfLValue(LV, Loc).getScalarVal(); + return RValue::get(EmitPointerAuthUnqualify(PtrAuth, Value, LV.getType(), + LV.getAddress(), + /*known nonnull*/ false)); + } + if (LV.isObjCWeak()) { // load of a __weak object. Address AddrWeakObj = LV.getAddress(); @@ -2490,6 +2507,13 @@ void CodeGenFunction::EmitStoreThroughLValue(RValue Src, LValue Dst, return EmitStoreThroughBitfieldLValue(Src, Dst); } + // Handle __ptrauth qualification by re-signing the value. + if (PointerAuthQualifier PointerAuth = Dst.getQuals().getPointerAuth()) { + Src = RValue::get(EmitPointerAuthQualify(PointerAuth, Src.getScalarVal(), + Dst.getType(), Dst.getAddress(), + /*known nonnull*/ false)); + } + // There's special magic for assigning into an ARC-qualified l-value. if (Qualifiers::ObjCLifetime Lifetime = Dst.getQuals().getObjCLifetime()) { switch (Lifetime) { @@ -5792,6 +5816,28 @@ CGCallee CodeGenFunction::EmitCallee(const Expr *E) { return EmitCallee(ICE->getSubExpr()); } + // Try to remember the original __ptrauth qualifier for loads of + // function pointers. + if (ICE->getCastKind() == CK_LValueToRValue) { + const Expr *SubExpr = ICE->getSubExpr(); + if (const auto *PtrType = SubExpr->getType()->getAs()) { + std::pair Result = + EmitOrigPointerRValue(E); + + QualType FunctionType = PtrType->getPointeeType(); + assert(FunctionType->isFunctionType()); + + GlobalDecl GD; + if (const auto *VD = + dyn_cast_or_null(E->getReferencedDeclOfCallee())) { + GD = GlobalDecl(VD); + } + CGCalleeInfo CalleeInfo(FunctionType->getAs(), GD); + CGCallee Callee(CalleeInfo, Result.first, Result.second); + return Callee; + } + } + // Resolve direct calls. } else if (auto DRE = dyn_cast(E)) { if (auto FD = dyn_cast(DRE->getDecl())) { @@ -5854,6 +5900,18 @@ LValue CodeGenFunction::EmitBinaryOperatorLValue(const BinaryOperator *E) { switch (getEvaluationKind(E->getType())) { case TEK_Scalar: { + if (PointerAuthQualifier PtrAuth = + E->getLHS()->getType().getPointerAuth()) { + LValue LV = EmitCheckedLValue(E->getLHS(), TCK_Store); + LValue CopiedLV = LV; + CopiedLV.getQuals().removePointerAuth(); + llvm::Value *RV = + EmitPointerAuthQualify(PtrAuth, E->getRHS(), CopiedLV.getAddress()); + EmitNullabilityCheck(CopiedLV, RV, E->getExprLoc()); + EmitStoreThroughLValue(RValue::get(RV), CopiedLV); + return LV; + } + switch (E->getLHS()->getType().getObjCLifetime()) { case Qualifiers::OCL_Strong: return EmitARCStoreStrong(E, /*ignored*/ false).first; diff --git a/clang/lib/CodeGen/CGExprConstant.cpp b/clang/lib/CodeGen/CGExprConstant.cpp index b016c6e36d1a8..b21ebeee4bed1 100644 --- a/clang/lib/CodeGen/CGExprConstant.cpp +++ b/clang/lib/CodeGen/CGExprConstant.cpp @@ -2049,10 +2049,13 @@ namespace { struct ConstantLValue { llvm::Constant *Value; bool HasOffsetApplied; + bool HasDestPointerAuth; /*implicit*/ ConstantLValue(llvm::Constant *value, - bool hasOffsetApplied = false) - : Value(value), HasOffsetApplied(hasOffsetApplied) {} + bool hasOffsetApplied = false, + bool hasDestPointerAuth = false) + : Value(value), HasOffsetApplied(hasOffsetApplied), + HasDestPointerAuth(hasDestPointerAuth) {} /*implicit*/ ConstantLValue(ConstantAddress address) : ConstantLValue(address.getPointer()) {} @@ -2157,6 +2160,14 @@ llvm::Constant *ConstantLValueEmitter::tryEmit() { value = applyOffset(value); } + // Apply pointer-auth signing from the destination type. + if (PointerAuthQualifier PointerAuth = DestType.getPointerAuth(); + PointerAuth && !result.HasDestPointerAuth) { + value = Emitter.tryEmitConstantSignedPointer(value, PointerAuth); + if (!value) + return nullptr; + } + // Convert to the appropriate type; this could be an lvalue for // an integer. FIXME: performAddrSpaceCast if (isa(destTy)) @@ -2200,6 +2211,12 @@ ConstantLValueEmitter::tryEmitBase(const APValue::LValueBase &base) { return CGM.GetWeakRefReference(D).getPointer(); auto PtrAuthSign = [&](llvm::Constant *C) { + if (PointerAuthQualifier PointerAuth = DestType.getPointerAuth()) { + C = applyOffset(C); + C = Emitter.tryEmitConstantSignedPointer(C, PointerAuth); + return ConstantLValue(C, /*applied offset*/ true, /*signed*/ true); + } + CGPointerAuthInfo AuthInfo; if (EnablePtrAuthFunctionTypeDiscrimination) @@ -2213,7 +2230,7 @@ ConstantLValueEmitter::tryEmitBase(const APValue::LValueBase &base) { C = CGM.getConstantSignedPointer( C, AuthInfo.getKey(), nullptr, cast_or_null(AuthInfo.getDiscriminator())); - return ConstantLValue(C, /*applied offset*/ true); + return ConstantLValue(C, /*applied offset*/ true, /*signed*/ true); } return ConstantLValue(C); diff --git a/clang/lib/CodeGen/CGExprScalar.cpp b/clang/lib/CodeGen/CGExprScalar.cpp index e9a7ba509350c..8dbbcdaef25d8 100644 --- a/clang/lib/CodeGen/CGExprScalar.cpp +++ b/clang/lib/CodeGen/CGExprScalar.cpp @@ -2261,6 +2261,53 @@ Value *ScalarExprEmitter::VisitInitListExpr(InitListExpr *E) { return V; } +static bool isDeclRefKnownNonNull(CodeGenFunction &CGF, const ValueDecl *D) { + return !D->isWeak(); +} + +static bool isLValueKnownNonNull(CodeGenFunction &CGF, const Expr *E) { + E = E->IgnoreParens(); + + if (const auto *UO = dyn_cast(E)) + if (UO->getOpcode() == UO_Deref) + return CGF.isPointerKnownNonNull(UO->getSubExpr()); + + if (const auto *DRE = dyn_cast(E)) + return isDeclRefKnownNonNull(CGF, DRE->getDecl()); + + if (const auto *ME = dyn_cast(E)) { + if (isa(ME->getMemberDecl())) + return true; + return isDeclRefKnownNonNull(CGF, ME->getMemberDecl()); + } + + // Array subscripts? Anything else? + + return false; +} + +bool CodeGenFunction::isPointerKnownNonNull(const Expr *E) { + assert(E->getType()->isSignableType()); + + E = E->IgnoreParens(); + + if (isa(E)) + return true; + + if (const auto *UO = dyn_cast(E)) + if (UO->getOpcode() == UO_AddrOf) + return isLValueKnownNonNull(*this, UO->getSubExpr()); + + if (const auto *CE = dyn_cast(E)) + if (CE->getCastKind() == CK_FunctionToPointerDecay || + CE->getCastKind() == CK_ArrayToPointerDecay) + return isLValueKnownNonNull(*this, CE->getSubExpr()); + + // Maybe honor __nonnull? + + return false; +} + bool CodeGenFunction::ShouldNullCheckClassCastValue(const CastExpr *CE) { const Expr *E = CE->getSubExpr(); @@ -4985,6 +5032,21 @@ Value *ScalarExprEmitter::VisitBinAssign(const BinaryOperator *E) { Value *RHS; LValue LHS; + if (PointerAuthQualifier PtrAuth = E->getLHS()->getType().getPointerAuth()) { + LValue LV = CGF.EmitCheckedLValue(E->getLHS(), CodeGenFunction::TCK_Store); + LV.getQuals().removePointerAuth(); + llvm::Value *RV = + CGF.EmitPointerAuthQualify(PtrAuth, E->getRHS(), LV.getAddress()); + CGF.EmitNullabilityCheck(LV, RV, E->getExprLoc()); + CGF.EmitStoreThroughLValue(RValue::get(RV), LV); + + if (Ignore) + return nullptr; + RV = CGF.EmitPointerAuthUnqualify(PtrAuth, RV, LV.getType(), + LV.getAddress(), /*nonnull*/ false); + return RV; + } + switch (E->getLHS()->getType().getObjCLifetime()) { case Qualifiers::OCL_Strong: std::tie(LHS, RHS) = CGF.EmitARCStoreStrong(E, Ignore); diff --git a/clang/lib/CodeGen/CGPointerAuth.cpp b/clang/lib/CodeGen/CGPointerAuth.cpp index 4b032306ead72..0a183a8524c17 100644 --- a/clang/lib/CodeGen/CGPointerAuth.cpp +++ b/clang/lib/CodeGen/CGPointerAuth.cpp @@ -125,6 +125,33 @@ CGPointerAuthInfo CodeGenFunction::EmitPointerAuthInfo( Schema.authenticatesNullValues(), Discriminator); } +CGPointerAuthInfo +CodeGenFunction::EmitPointerAuthInfo(PointerAuthQualifier Qual, + Address StorageAddress) { + assert(Qual && "don't call this if you don't know that the Qual is present"); + if (Qual.hasKeyNone()) + return CGPointerAuthInfo(); + + llvm::Value *Discriminator = nullptr; + if (unsigned Extra = Qual.getExtraDiscriminator()) + Discriminator = llvm::ConstantInt::get(IntPtrTy, Extra); + + if (Qual.isAddressDiscriminated()) { + assert(StorageAddress.isValid() && + "address discrimination without address"); + llvm::Value *StoragePtr = StorageAddress.emitRawPointer(*this); + if (Discriminator) + Discriminator = + EmitPointerAuthBlendDiscriminator(StoragePtr, Discriminator); + else + Discriminator = Builder.CreatePtrToInt(StoragePtr, IntPtrTy); + } + + return CGPointerAuthInfo(Qual.getKey(), Qual.getAuthenticationMode(), + Qual.isIsaPointer(), Qual.authenticatesNullValues(), + Discriminator); +} + /// Return the natural pointer authentication for values of the given /// pointee type. static CGPointerAuthInfo @@ -166,6 +193,91 @@ CGPointerAuthInfo CodeGenModule::getPointerAuthInfoForType(QualType T) { return ::getPointerAuthInfoForType(*this, T); } +static std::pair +emitLoadOfOrigPointerRValue(CodeGenFunction &CGF, const LValue &LV, + SourceLocation Loc) { + llvm::Value *Value = CGF.EmitLoadOfScalar(LV, Loc); + CGPointerAuthInfo AuthInfo; + if (PointerAuthQualifier PtrAuth = LV.getQuals().getPointerAuth()) + AuthInfo = CGF.EmitPointerAuthInfo(PtrAuth, LV.getAddress()); + else + AuthInfo = getPointerAuthInfoForType(CGF.CGM, LV.getType()); + return {Value, AuthInfo}; +} + +/// Retrieve a pointer rvalue and its ptrauth info. When possible, avoid +/// needlessly resigning the pointer. +std::pair +CodeGenFunction::EmitOrigPointerRValue(const Expr *E) { + assert(E->getType()->isSignableType()); + + E = E->IgnoreParens(); + if (const auto *Load = dyn_cast(E)) { + if (Load->getCastKind() == CK_LValueToRValue) { + E = Load->getSubExpr()->IgnoreParens(); + + // We're semantically required to not emit loads of certain DREs naively. + if (const auto *RefExpr = dyn_cast(E)) { + if (ConstantEmission Result = tryEmitAsConstant(RefExpr)) { + // Fold away a use of an intermediate variable. + if (!Result.isReference()) + return {Result.getValue(), + getPointerAuthInfoForType(CGM, RefExpr->getType())}; + + // Fold away a use of an intermediate reference. + LValue LV = Result.getReferenceLValue(*this, RefExpr); + return emitLoadOfOrigPointerRValue(*this, LV, RefExpr->getLocation()); + } + } + + // Otherwise, load and use the pointer + LValue LV = EmitCheckedLValue(E, CodeGenFunction::TCK_Load); + return emitLoadOfOrigPointerRValue(*this, LV, E->getExprLoc()); + } + } + + // Fallback: just use the normal rules for the type. + llvm::Value *Value = EmitScalarExpr(E); + return {Value, getPointerAuthInfoForType(CGM, E->getType())}; +} + +llvm::Value * +CodeGenFunction::EmitPointerAuthQualify(PointerAuthQualifier DestQualifier, + const Expr *E, + Address DestStorageAddress) { + assert(DestQualifier); + auto [Value, CurAuthInfo] = EmitOrigPointerRValue(E); + + CGPointerAuthInfo DestAuthInfo = + EmitPointerAuthInfo(DestQualifier, DestStorageAddress); + return emitPointerAuthResign(Value, E->getType(), CurAuthInfo, DestAuthInfo, + isPointerKnownNonNull(E)); +} + +llvm::Value *CodeGenFunction::EmitPointerAuthQualify( + PointerAuthQualifier DestQualifier, llvm::Value *Value, + QualType PointerType, Address DestStorageAddress, bool IsKnownNonNull) { + assert(DestQualifier); + + CGPointerAuthInfo CurAuthInfo = getPointerAuthInfoForType(CGM, PointerType); + CGPointerAuthInfo DestAuthInfo = + EmitPointerAuthInfo(DestQualifier, DestStorageAddress); + return emitPointerAuthResign(Value, PointerType, CurAuthInfo, DestAuthInfo, + IsKnownNonNull); +} + +llvm::Value *CodeGenFunction::EmitPointerAuthUnqualify( + PointerAuthQualifier CurQualifier, llvm::Value *Value, QualType PointerType, + Address CurStorageAddress, bool IsKnownNonNull) { + assert(CurQualifier); + + CGPointerAuthInfo CurAuthInfo = + EmitPointerAuthInfo(CurQualifier, CurStorageAddress); + CGPointerAuthInfo DestAuthInfo = getPointerAuthInfoForType(CGM, PointerType); + return emitPointerAuthResign(Value, PointerType, CurAuthInfo, DestAuthInfo, + IsKnownNonNull); +} + static bool isZeroConstant(const llvm::Value *Value) { if (const auto *CI = dyn_cast(Value)) return CI->isZero(); @@ -288,6 +400,23 @@ llvm::Value *CodeGenFunction::emitPointerAuthResign( return Value; } +void CodeGenFunction::EmitPointerAuthCopy(PointerAuthQualifier Qual, QualType T, + Address DestAddress, + Address SrcAddress) { + assert(Qual); + llvm::Value *Value = Builder.CreateLoad(SrcAddress); + + // If we're using address-discrimination, we have to re-sign the value. + if (Qual.isAddressDiscriminated()) { + CGPointerAuthInfo SrcPtrAuth = EmitPointerAuthInfo(Qual, SrcAddress); + CGPointerAuthInfo DestPtrAuth = EmitPointerAuthInfo(Qual, DestAddress); + Value = emitPointerAuthResign(Value, T, SrcPtrAuth, DestPtrAuth, + /*IsKnownNonNull=*/false); + } + + Builder.CreateStore(Value, DestAddress); +} + llvm::Constant * CodeGenModule::getConstantSignedPointer(llvm::Constant *Pointer, unsigned Key, llvm::Constant *StorageAddress, diff --git a/clang/lib/CodeGen/CodeGenFunction.h b/clang/lib/CodeGen/CodeGenFunction.h index aa07e5d6c8099..4c5e8a8a44926 100644 --- a/clang/lib/CodeGen/CodeGenFunction.h +++ b/clang/lib/CodeGen/CodeGenFunction.h @@ -4432,10 +4432,10 @@ class CodeGenFunction : public CodeGenTypeCache { } bool isReference() const { return ValueAndIsReference.getInt(); } - LValue getReferenceLValue(CodeGenFunction &CGF, Expr *refExpr) const { + LValue getReferenceLValue(CodeGenFunction &CGF, const Expr *RefExpr) const { assert(isReference()); return CGF.MakeNaturalAlignAddrLValue(ValueAndIsReference.getPointer(), - refExpr->getType()); + RefExpr->getType()); } llvm::Constant *getValue() const { @@ -4444,7 +4444,7 @@ class CodeGenFunction : public CodeGenTypeCache { } }; - ConstantEmission tryEmitAsConstant(DeclRefExpr *refExpr); + ConstantEmission tryEmitAsConstant(const DeclRefExpr *RefExpr); ConstantEmission tryEmitAsConstant(const MemberExpr *ME); llvm::Value *emitScalarConstant(const ConstantEmission &Constant, Expr *E); @@ -4588,6 +4588,26 @@ class CodeGenFunction : public CodeGenTypeCache { const CGPointerAuthInfo &Info, SmallVectorImpl &Bundles); + CGPointerAuthInfo EmitPointerAuthInfo(PointerAuthQualifier Qualifier, + Address StorageAddress); + llvm::Value *EmitPointerAuthQualify(PointerAuthQualifier Qualifier, + llvm::Value *Pointer, QualType ValueType, + Address StorageAddress, + bool IsKnownNonNull); + llvm::Value *EmitPointerAuthQualify(PointerAuthQualifier Qualifier, + const Expr *PointerExpr, + Address StorageAddress); + llvm::Value *EmitPointerAuthUnqualify(PointerAuthQualifier Qualifier, + llvm::Value *Pointer, + QualType PointerType, + Address StorageAddress, + bool IsKnownNonNull); + void EmitPointerAuthCopy(PointerAuthQualifier Qualifier, QualType Type, + Address DestField, Address SrcField); + + std::pair + EmitOrigPointerRValue(const Expr *E); + llvm::Value *authPointerToPointerCast(llvm::Value *ResultPtr, QualType SourceType, QualType DestType); Address authPointerToPointerCast(Address Ptr, QualType SourceType, diff --git a/clang/lib/Parse/ParseDecl.cpp b/clang/lib/Parse/ParseDecl.cpp index d77400e0f8272..8fa74ecff19aa 100644 --- a/clang/lib/Parse/ParseDecl.cpp +++ b/clang/lib/Parse/ParseDecl.cpp @@ -3400,6 +3400,45 @@ void Parser::DistributeCLateParsedAttrs(Decl *Dcl, } } +/// type-qualifier: +/// ('__ptrauth') '(' constant-expression +/// (',' constant-expression)[opt] +/// (',' constant-expression)[opt] ')' +void Parser::ParsePtrauthQualifier(ParsedAttributes &Attrs) { + assert(Tok.is(tok::kw___ptrauth)); + + IdentifierInfo *KwName = Tok.getIdentifierInfo(); + SourceLocation KwLoc = ConsumeToken(); + + BalancedDelimiterTracker T(*this, tok::l_paren); + if (T.expectAndConsume()) + return; + + ArgsVector ArgExprs; + do { + ExprResult ER = ParseAssignmentExpression(); + if (ER.isInvalid()) { + T.skipToEnd(); + return; + } + ArgExprs.push_back(ER.get()); + } while (TryConsumeToken(tok::comma)); + + T.consumeClose(); + SourceLocation EndLoc = T.getCloseLocation(); + + if (ArgExprs.empty() || ArgExprs.size() > 3) { + Diag(KwLoc, diag::err_ptrauth_qualifier_bad_arg_count); + return; + } + + Attrs.addNew(KwName, SourceRange(KwLoc, EndLoc), + /*scope*/ nullptr, SourceLocation(), ArgExprs.data(), + ArgExprs.size(), + ParsedAttr::Form::Keyword(/*IsAlignAs=*/false, + /*IsRegularKeywordAttribute=*/false)); +} + /// Bounds attributes (e.g., counted_by): /// AttrName '(' expression ')' void Parser::ParseBoundsAttribute(IdentifierInfo &AttrName, @@ -4267,6 +4306,11 @@ void Parser::ParseDeclarationSpecifiers( getLangOpts()); break; + // __ptrauth qualifier. + case tok::kw___ptrauth: + ParsePtrauthQualifier(DS.getAttributes()); + continue; + case tok::kw___sptr: case tok::kw___uptr: case tok::kw___ptr64: @@ -5980,6 +6024,7 @@ bool Parser::isTypeSpecifierQualifier() { case tok::kw___ptr32: case tok::kw___pascal: case tok::kw___unaligned: + case tok::kw___ptrauth: case tok::kw__Nonnull: case tok::kw__Nullable: @@ -6269,6 +6314,7 @@ bool Parser::isDeclarationSpecifier( case tok::kw___forceinline: case tok::kw___pascal: case tok::kw___unaligned: + case tok::kw___ptrauth: case tok::kw__Nonnull: case tok::kw__Nullable: @@ -6533,6 +6579,12 @@ void Parser::ParseTypeQualifierListOpt( ParseHLSLQualifiers(DS.getAttributes()); continue; + // __ptrauth qualifier. + case tok::kw___ptrauth: + ParsePtrauthQualifier(DS.getAttributes()); + EndLoc = PrevTokLocation; + continue; + case tok::kw___unaligned: isInvalid = DS.SetTypeQual(DeclSpec::TQ_unaligned, Loc, PrevSpec, DiagID, getLangOpts()); diff --git a/clang/lib/Sema/SemaCast.cpp b/clang/lib/Sema/SemaCast.cpp index 1591075ff05d8..14e16bc39eb3a 100644 --- a/clang/lib/Sema/SemaCast.cpp +++ b/clang/lib/Sema/SemaCast.cpp @@ -72,8 +72,11 @@ namespace { // Preceding an expression by a parenthesized type name converts the // value of the expression to the unqualified, non-atomic version of // the named type. + // Don't drop __ptrauth qualifiers. We want to treat casting to a + // __ptrauth-qualified type as an error instead of implicitly ignoring + // the qualifier. if (!S.Context.getLangOpts().ObjC && !DestType->isRecordType() && - !DestType->isArrayType()) { + !DestType->isArrayType() && !DestType.getPointerAuth()) { DestType = DestType.getAtomicUnqualifiedType(); } @@ -168,6 +171,14 @@ namespace { SrcExpr = src; } + void checkQualifiedDestType() { + // Destination type may not be qualified with __ptrauth. + if (DestType.getPointerAuth()) { + Self.Diag(DestRange.getBegin(), diag::err_ptrauth_qualifier_cast) + << DestType << DestRange; + } + } + /// Check for and handle non-overload placeholder expressions. void checkNonOverloadPlaceholders() { if (!isPlaceholder() || isPlaceholder(BuiltinType::Overload)) @@ -309,6 +320,8 @@ Sema::BuildCXXNamedCast(SourceLocation OpLoc, tok::TokenKind Kind, Op.OpRange = SourceRange(OpLoc, Parens.getEnd()); Op.DestRange = AngleBrackets; + Op.checkQualifiedDestType(); + switch (Kind) { default: llvm_unreachable("Unknown C++ cast!"); @@ -3412,6 +3425,8 @@ ExprResult Sema::BuildCStyleCastExpr(SourceLocation LPLoc, // -Wcast-qual DiagnoseCastQual(Op.Self, Op.SrcExpr, Op.DestType); + Op.checkQualifiedDestType(); + return Op.complete(CStyleCastExpr::Create( Context, Op.ResultType, Op.ValueKind, Op.Kind, Op.SrcExpr.get(), &Op.BasePath, CurFPFeatureOverrides(), CastTypeInfo, LPLoc, RPLoc)); @@ -3431,6 +3446,8 @@ ExprResult Sema::BuildCXXFunctionalCastExpr(TypeSourceInfo *CastTypeInfo, if (Op.SrcExpr.isInvalid()) return ExprError(); + Op.checkQualifiedDestType(); + auto *SubExpr = Op.SrcExpr.get(); if (auto *BindExpr = dyn_cast(SubExpr)) SubExpr = BindExpr->getSubExpr(); diff --git a/clang/lib/Sema/SemaChecking.cpp b/clang/lib/Sema/SemaChecking.cpp index d0143d29a4bcc..42da9ba97e0d3 100644 --- a/clang/lib/Sema/SemaChecking.cpp +++ b/clang/lib/Sema/SemaChecking.cpp @@ -1550,6 +1550,48 @@ bool Sema::checkConstantPointerAuthKey(Expr *Arg, unsigned &Result) { return false; } +bool Sema::checkPointerAuthDiscriminatorArg(Expr *Arg, + PointerAuthDiscArgKind Kind, + unsigned &IntVal) { + if (!Arg) { + IntVal = 0; + return true; + } + + std::optional Result = Arg->getIntegerConstantExpr(Context); + if (!Result) { + Diag(Arg->getExprLoc(), diag::err_ptrauth_arg_not_ice); + return false; + } + + unsigned Max; + bool IsAddrDiscArg = false; + + switch (Kind) { + case PADAK_AddrDiscPtrAuth: + Max = 1; + IsAddrDiscArg = true; + break; + case PADAK_ExtraDiscPtrAuth: + Max = PointerAuthQualifier::MaxDiscriminator; + break; + }; + + if (*Result < 0 || *Result > Max) { + if (IsAddrDiscArg) + Diag(Arg->getExprLoc(), diag::err_ptrauth_address_discrimination_invalid) + << Result->getExtValue(); + else + Diag(Arg->getExprLoc(), diag::err_ptrauth_extra_discriminator_invalid) + << Result->getExtValue() << Max; + + return false; + }; + + IntVal = Result->getZExtValue(); + return true; +} + static std::pair findConstantBaseAndOffset(Sema &S, Expr *E) { // Must evaluate as a pointer. @@ -3957,6 +3999,14 @@ ExprResult Sema::BuildAtomicExpr(SourceRange CallRange, SourceRange ExprRange, ValType = AtomTy; } + PointerAuthQualifier PointerAuth = AtomTy.getPointerAuth(); + if (PointerAuth && PointerAuth.isAddressDiscriminated()) { + Diag(ExprRange.getBegin(), + diag::err_atomic_op_needs_non_address_discriminated_pointer) + << 0 << Ptr->getType() << Ptr->getSourceRange(); + return ExprError(); + } + // For an arithmetic operation, the implied arithmetic must be well-formed. if (Form == Arithmetic) { // GCC does not enforce these rules for GNU atomics, but we do to help catch @@ -4329,6 +4379,13 @@ ExprResult Sema::BuiltinAtomicOverloaded(ExprResult TheCallResult) { << FirstArg->getType() << 0 << FirstArg->getSourceRange(); return ExprError(); } + PointerAuthQualifier PointerAuth = ValType.getPointerAuth(); + if (PointerAuth && PointerAuth.isAddressDiscriminated()) { + Diag(FirstArg->getBeginLoc(), + diag::err_atomic_op_needs_non_address_discriminated_pointer) + << 1 << ValType << FirstArg->getSourceRange(); + return ExprError(); + } if (ValType.isConstQualified()) { Diag(DRE->getBeginLoc(), diag::err_atomic_builtin_cannot_be_const) diff --git a/clang/lib/Sema/SemaDecl.cpp b/clang/lib/Sema/SemaDecl.cpp index 240ce5391af81..5f811c824e11d 100644 --- a/clang/lib/Sema/SemaDecl.cpp +++ b/clang/lib/Sema/SemaDecl.cpp @@ -15454,6 +15454,12 @@ ParmVarDecl *Sema::CheckParameter(DeclContext *DC, SourceLocation StartLoc, New->setType(T); } + // __ptrauth is forbidden on parameters. + if (T.getPointerAuth()) { + Diag(NameLoc, diag::err_ptrauth_qualifier_invalid) << T << 1; + New->setInvalidDecl(); + } + // ISO/IEC TR 18037 S6.7.3: "The type of an object with automatic storage // duration shall not be qualified by an address-space qualifier." // Since all parameters have automatic store duration, they can not have @@ -19456,9 +19462,14 @@ void Sema::ActOnFields(Scope *S, SourceLocation RecLoc, Decl *EnclosingDecl, RecordArgPassingKind::CanNeverPassInRegs) Record->setArgPassingRestrictions( RecordArgPassingKind::CanNeverPassInRegs); - } else if (FT.getQualifiers().getObjCLifetime() == Qualifiers::OCL_Weak) + } else if (FT.getQualifiers().getObjCLifetime() == Qualifiers::OCL_Weak) { Record->setArgPassingRestrictions( RecordArgPassingKind::CanNeverPassInRegs); + } else if (PointerAuthQualifier Q = FT.getPointerAuth(); + Q && Q.isAddressDiscriminated()) { + Record->setArgPassingRestrictions( + RecordArgPassingKind::CanNeverPassInRegs); + } } if (Record && FD->getType().isVolatileQualified()) diff --git a/clang/lib/Sema/SemaDeclCXX.cpp b/clang/lib/Sema/SemaDeclCXX.cpp index 74f925f18560a..2247aded9384a 100644 --- a/clang/lib/Sema/SemaDeclCXX.cpp +++ b/clang/lib/Sema/SemaDeclCXX.cpp @@ -9471,6 +9471,8 @@ struct SpecialMemberDeletionInfo bool shouldDeleteForVariantObjCPtrMember(FieldDecl *FD, QualType FieldType); + bool shouldDeleteForVariantPtrAuthMember(const FieldDecl *FD); + bool visitBase(CXXBaseSpecifier *Base) { return shouldDeleteForBase(Base); } bool visitField(FieldDecl *Field) { return shouldDeleteForField(Field); } @@ -9639,6 +9641,30 @@ bool SpecialMemberDeletionInfo::shouldDeleteForVariantObjCPtrMember( return true; } +bool SpecialMemberDeletionInfo::shouldDeleteForVariantPtrAuthMember( + const FieldDecl *FD) { + QualType FieldType = S.Context.getBaseElementType(FD->getType()); + // Copy/move constructors/assignment operators are deleted if the field has an + // address-discriminated ptrauth qualifier. + PointerAuthQualifier Q = FieldType.getPointerAuth(); + + if (!Q || !Q.isAddressDiscriminated()) + return false; + + if (CSM == CXXSpecialMemberKind::DefaultConstructor || + CSM == CXXSpecialMemberKind::Destructor) + return false; + + if (Diagnose) { + auto *ParentClass = cast(FD->getParent()); + S.Diag(FD->getLocation(), diag::note_deleted_special_member_class_subobject) + << llvm::to_underlying(getEffectiveCSM()) << ParentClass + << /*IsField*/ true << FD << 4 << /*IsDtorCallInCtor*/ false << 2; + } + + return true; +} + /// Check whether we should delete a special member function due to the class /// having a particular direct or virtual base class. bool SpecialMemberDeletionInfo::shouldDeleteForBase(CXXBaseSpecifier *Base) { @@ -9677,6 +9703,9 @@ bool SpecialMemberDeletionInfo::shouldDeleteForField(FieldDecl *FD) { if (inUnion() && shouldDeleteForVariantObjCPtrMember(FD, FieldType)) return true; + if (inUnion() && shouldDeleteForVariantPtrAuthMember(FD)) + return true; + if (CSM == CXXSpecialMemberKind::DefaultConstructor) { // For a default constructor, all references must be initialized in-class // and, if a union, it must have a non-const member. @@ -9740,6 +9769,9 @@ bool SpecialMemberDeletionInfo::shouldDeleteForField(FieldDecl *FD) { if (shouldDeleteForVariantObjCPtrMember(&*UI, UnionFieldType)) return true; + if (shouldDeleteForVariantPtrAuthMember(&*UI)) + return true; + if (!UnionFieldType.isConstQualified()) AllVariantFieldsAreConst = false; @@ -10589,6 +10621,12 @@ void Sema::checkIllFormedTrivialABIStruct(CXXRecordDecl &RD) { return; } + // Ill-formed if the field is an address-discriminated pointer. + if (FT.hasAddressDiscriminatedPointerAuth()) { + PrintDiagAndRemoveAttr(6); + return; + } + if (const auto *RT = FT->getBaseElementTypeUnsafe()->getAs()) if (!RT->isDependentType() && !cast(RT->getDecl())->canPassInRegisters()) { diff --git a/clang/lib/Sema/SemaExpr.cpp b/clang/lib/Sema/SemaExpr.cpp index c25daaa022f49..3ac7d61546ceb 100644 --- a/clang/lib/Sema/SemaExpr.cpp +++ b/clang/lib/Sema/SemaExpr.cpp @@ -8107,6 +8107,13 @@ static QualType checkConditionalPointerCompatibility(Sema &S, ExprResult &LHS, lhQual.removeCVRQualifiers(); rhQual.removeCVRQualifiers(); + if (!lhQual.getPointerAuth().isEquivalent(rhQual.getPointerAuth())) { + S.Diag(Loc, diag::err_typecheck_cond_incompatible_ptrauth) + << LHSTy << RHSTy << LHS.get()->getSourceRange() + << RHS.get()->getSourceRange(); + return QualType(); + } + // OpenCL v2.0 specification doesn't extend compatibility of type qualifiers // (C99 6.7.3) for address spaces. We assume that the check should behave in // the same manner as it's defined for CVR qualifiers, so for OpenCL two @@ -9027,6 +9034,10 @@ checkPointerTypesForAssignment(Sema &S, QualType LHSType, QualType RHSType, else if (lhq.getObjCLifetime() != rhq.getObjCLifetime()) ConvTy = Sema::IncompatiblePointerDiscardsQualifiers; + // Treat pointer-auth mismatches as fatal. + else if (!lhq.getPointerAuth().isEquivalent(rhq.getPointerAuth())) + ConvTy = Sema::IncompatiblePointerDiscardsQualifiers; + // For GCC/MS compatibility, other qualifier mismatches are treated // as still compatible in C. else ConvTy = Sema::CompatiblePointerDiscardsQualifiers; @@ -17025,6 +17036,9 @@ bool Sema::DiagnoseAssignmentResult(AssignConvertType ConvTy, } else if (lhq.getObjCLifetime() != rhq.getObjCLifetime()) { DiagKind = diag::err_typecheck_incompatible_ownership; break; + } else if (!lhq.getPointerAuth().isEquivalent(rhq.getPointerAuth())) { + DiagKind = diag::err_typecheck_incompatible_ptrauth; + break; } llvm_unreachable("unknown error case for discarding qualifiers!"); diff --git a/clang/lib/Sema/SemaExprCXX.cpp b/clang/lib/Sema/SemaExprCXX.cpp index dfb5824a1c3d7..8df590fa624cf 100644 --- a/clang/lib/Sema/SemaExprCXX.cpp +++ b/clang/lib/Sema/SemaExprCXX.cpp @@ -7770,6 +7770,11 @@ QualType Sema::FindCompositePointerType(SourceLocation Loc, else return QualType(); + if (Q1.getPointerAuth().isEquivalent(Q2.getPointerAuth())) + Quals.setPointerAuth(Q1.getPointerAuth()); + else + return QualType(); + Steps.back().Quals = Quals; if (Q1 != Quals || Q2 != Quals) NeedConstBefore = Steps.size() - 1; diff --git a/clang/lib/Sema/SemaObjCProperty.cpp b/clang/lib/Sema/SemaObjCProperty.cpp index 6db2c246de791..f37982eddace9 100644 --- a/clang/lib/Sema/SemaObjCProperty.cpp +++ b/clang/lib/Sema/SemaObjCProperty.cpp @@ -180,6 +180,9 @@ Decl *SemaObjC::ActOnProperty(Scope *S, SourceLocation AtLoc, 0); TypeSourceInfo *TSI = SemaRef.GetTypeForDeclarator(FD.D); QualType T = TSI->getType(); + if (T.getPointerAuth().isPresent()) { + Diag(AtLoc, diag::err_ptrauth_qualifier_invalid) << T << 2; + } if (!getOwnershipRule(Attributes)) { Attributes |= deducePropertyOwnershipFromType(SemaRef, T); } diff --git a/clang/lib/Sema/SemaOverload.cpp b/clang/lib/Sema/SemaOverload.cpp index 9c2df0b21d278..55634aa75ae25 100644 --- a/clang/lib/Sema/SemaOverload.cpp +++ b/clang/lib/Sema/SemaOverload.cpp @@ -3709,6 +3709,10 @@ static bool isQualificationConversionStep(QualType FromType, QualType ToType, ToQuals.removeObjCGCAttr(); } + // __ptrauth qualifiers must match exactly. + if (FromQuals.getPointerAuth() != ToQuals.getPointerAuth()) + return false; + // -- for every j > 0, if const is in cv 1,j then const is in cv // 2,j, and similarly for volatile. if (!CStyle && !ToQuals.compatiblyIncludes(FromQuals, Ctx)) @@ -11467,6 +11471,17 @@ static void DiagnoseBadConversion(Sema &S, OverloadCandidate *Cand, return; } + if (!FromQs.getPointerAuth().isEquivalent(ToQs.getPointerAuth())) { + S.Diag(Fn->getLocation(), diag::note_ovl_candidate_bad_ptrauth) + << (unsigned)FnKindPair.first << (unsigned)FnKindPair.second << FnDesc + << FromTy << !!FromQs.getPointerAuth() + << FromQs.getPointerAuth().getAsString() << !!ToQs.getPointerAuth() + << ToQs.getPointerAuth().getAsString() << I + 1 + << (FromExpr ? FromExpr->getSourceRange() : SourceRange()); + MaybeEmitInheritedConstructorNote(S, Cand->FoundDecl); + return; + } + unsigned CVR = FromQs.getCVRQualifiers() & ~ToQs.getCVRQualifiers(); assert(CVR && "expected qualifiers mismatch"); diff --git a/clang/lib/Sema/SemaType.cpp b/clang/lib/Sema/SemaType.cpp index 33b1d8ca4dfa0..eba7267904fb2 100644 --- a/clang/lib/Sema/SemaType.cpp +++ b/clang/lib/Sema/SemaType.cpp @@ -2552,6 +2552,12 @@ bool Sema::CheckFunctionReturnType(QualType T, SourceLocation Loc) { return true; } + // __ptrauth is illegal on a function return type. + if (T.getPointerAuth()) { + Diag(Loc, diag::err_ptrauth_qualifier_invalid) << T << 0; + return true; + } + if (T.hasNonTrivialToPrimitiveDestructCUnion() || T.hasNonTrivialToPrimitiveCopyCUnion()) checkNonTrivialCUnion(T, Loc, NTCUC_FunctionReturn, @@ -2657,6 +2663,10 @@ QualType Sema::BuildFunctionType(QualType T, } else if (ParamType->isWebAssemblyTableType()) { Diag(Loc, diag::err_wasm_table_as_function_parameter); Invalid = true; + } else if (ParamType.getPointerAuth()) { + // __ptrauth is illegal on a function return type. + Diag(Loc, diag::err_ptrauth_qualifier_invalid) << T << 1; + Invalid = true; } // C++2a [dcl.fct]p4: @@ -4974,6 +4984,11 @@ static TypeSourceInfo *GetFullTypeForDeclarator(TypeProcessingState &state, } } + // __ptrauth is illegal on a function return type. + if (T.getPointerAuth()) { + S.Diag(DeclType.Loc, diag::err_ptrauth_qualifier_invalid) << T << 0; + } + if (LangOpts.OpenCL) { // OpenCL v2.0 s6.12.5 - A block cannot be the return value of a // function. @@ -8333,6 +8348,65 @@ static void HandleNeonVectorTypeAttr(QualType &CurType, const ParsedAttr &Attr, CurType = S.Context.getVectorType(CurType, numElts, VecKind); } +/// Handle the __ptrauth qualifier. +static void HandlePtrAuthQualifier(ASTContext &Ctx, QualType &T, + const ParsedAttr &Attr, Sema &S) { + + assert((Attr.getNumArgs() > 0 && Attr.getNumArgs() <= 3) && + "__ptrauth qualifier takes between 1 and 3 arguments"); + Expr *KeyArg = Attr.getArgAsExpr(0); + Expr *IsAddressDiscriminatedArg = + Attr.getNumArgs() >= 2 ? Attr.getArgAsExpr(1) : nullptr; + Expr *ExtraDiscriminatorArg = + Attr.getNumArgs() >= 3 ? Attr.getArgAsExpr(2) : nullptr; + + unsigned Key; + if (S.checkConstantPointerAuthKey(KeyArg, Key)) { + Attr.setInvalid(); + return; + } + assert(Key <= PointerAuthQualifier::MaxKey && "ptrauth key is out of range"); + + bool IsInvalid = false; + unsigned IsAddressDiscriminated, ExtraDiscriminator; + IsInvalid |= !S.checkPointerAuthDiscriminatorArg(IsAddressDiscriminatedArg, + Sema::PADAK_AddrDiscPtrAuth, + IsAddressDiscriminated); + IsInvalid |= !S.checkPointerAuthDiscriminatorArg( + ExtraDiscriminatorArg, Sema::PADAK_ExtraDiscPtrAuth, ExtraDiscriminator); + + if (IsInvalid) { + Attr.setInvalid(); + return; + } + + if (!T->isSignableType() && !T->isDependentType()) { + S.Diag(Attr.getLoc(), diag::err_ptrauth_qualifier_nonpointer) << T; + Attr.setInvalid(); + return; + } + + if (T.getPointerAuth()) { + S.Diag(Attr.getLoc(), diag::err_ptrauth_qualifier_redundant) + << T << Attr.getAttrName()->getName(); + Attr.setInvalid(); + return; + } + + if (!S.getLangOpts().PointerAuthIntrinsics) { + S.Diag(Attr.getLoc(), diag::err_ptrauth_disabled) << Attr.getRange(); + Attr.setInvalid(); + return; + } + + assert((!IsAddressDiscriminatedArg || IsAddressDiscriminated <= 1) && + "address discriminator arg should be either 0 or 1"); + PointerAuthQualifier Qual = PointerAuthQualifier::Create( + Key, IsAddressDiscriminated, ExtraDiscriminator, + PointerAuthenticationMode::SignAndAuth, false, false); + T = S.Context.getPointerAuthType(T, Qual); +} + /// HandleArmSveVectorBitsTypeAttr - The "arm_sve_vector_bits" attribute is /// used to create fixed-length versions of sizeless SVE types defined by /// the ACLE, such as svint32_t and svbool_t. @@ -8788,6 +8862,11 @@ static void processTypeAttrs(TypeProcessingState &state, QualType &type, HandleOpenCLAccessAttr(type, attr, state.getSema()); attr.setUsedAsTypeAttr(); break; + case ParsedAttr::AT_PointerAuth: + HandlePtrAuthQualifier(state.getSema().Context, type, attr, + state.getSema()); + attr.setUsedAsTypeAttr(); + break; case ParsedAttr::AT_LifetimeBound: if (TAL == TAL_DeclChunk) HandleLifetimeBoundAttr(state, type, attr); diff --git a/clang/lib/Sema/TreeTransform.h b/clang/lib/Sema/TreeTransform.h index bb58ec49612c8..2469991bf2ce8 100644 --- a/clang/lib/Sema/TreeTransform.h +++ b/clang/lib/Sema/TreeTransform.h @@ -5287,6 +5287,17 @@ QualType TreeTransform::RebuildQualifiedType(QualType T, return QualType(); } + PointerAuthQualifier LocalPointerAuth = Quals.getPointerAuth(); + if (LocalPointerAuth.isPresent()) { + if (T.getPointerAuth().isPresent()) { + SemaRef.Diag(Loc, diag::err_ptrauth_qualifier_redundant) + << TL.getType() << "__ptrauth"; + return QualType(); + } else if (!T->isSignableType() && !T->isDependentType()) { + SemaRef.Diag(Loc, diag::err_ptrauth_qualifier_nonpointer) << T; + return QualType(); + } + } // C++ [dcl.fct]p7: // [When] adding cv-qualifications on top of the function type [...] the // cv-qualifiers are ignored. diff --git a/clang/test/AST/ast-dump-ptrauth-json.cpp b/clang/test/AST/ast-dump-ptrauth-json.cpp index 125cda0cff53a..8526598c491c1 100644 --- a/clang/test/AST/ast-dump-ptrauth-json.cpp +++ b/clang/test/AST/ast-dump-ptrauth-json.cpp @@ -1,5 +1,8 @@ // RUN: %clang_cc1 -triple arm64-apple-ios -fptrauth-calls -fptrauth-intrinsics -std=c++11 -ast-dump=json %s | FileCheck %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -fptrauth-calls -fptrauth-intrinsics -std=c++11 -ast-dump=json %s | FileCheck %s // CHECK: "name": "__builtin_ptrauth_type_discriminator", +// CHECK: "qualType": "int *__ptrauth(1,1,123)" int d = __builtin_ptrauth_type_discriminator(int()); +int * __ptrauth(1,1,123) p; diff --git a/clang/test/CodeGen/ptrauth-debuginfo.c b/clang/test/CodeGen/ptrauth-debuginfo.c new file mode 100644 index 0000000000000..b76baffadd9a1 --- /dev/null +++ b/clang/test/CodeGen/ptrauth-debuginfo.c @@ -0,0 +1,35 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios \ +// RUN: -fptrauth-calls -fptrauth-intrinsics -emit-llvm -fblocks \ +// RUN: %s -debug-info-kind=limited -o - | FileCheck %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu \ +// RUN: -fptrauth-calls -fptrauth-intrinsics -emit-llvm -fblocks \ +// RUN: %s -debug-info-kind=limited -o - | FileCheck %s + +// Constant initializers for data pointers. +extern int external_int; + +int *__ptrauth(1, 0, 1234) g1 = &external_int; +// CHECK: !DIDerivedType(tag: DW_TAG_LLVM_ptrauth_type, +// CHECK-SAME: ptrAuthKey: 1, +// CHECK-SAME: ptrAuthIsAddressDiscriminated: false, +// CHECK-SAME: ptrAuthExtraDiscriminator: 1234, +// CHECK-SAME: ptrAuthIsaPointer: false, +// CHECK-SAME: ptrAuthAuthenticatesNullValues: false) + +struct A { + int value; +}; +struct A *createA(void); + +void f() { + __block struct A *__ptrauth(0, 1, 1236) ptr = createA(); + ^{ + (void)ptr->value; + }(); +} +// CHECK: !DIDerivedType(tag: DW_TAG_LLVM_ptrauth_type, +// CHECK-NOT: ptrAuthKey +// CHECK-SAME: ptrAuthIsAddressDiscriminated: true, +// CHECK-SAME: ptrAuthExtraDiscriminator: 1236, +// CHECK-SAME: ptrAuthIsaPointer: false, +// CHECK-SAME: ptrAuthAuthenticatesNullValues: false) diff --git a/clang/test/CodeGen/ptrauth-qualifier-const-init.c b/clang/test/CodeGen/ptrauth-qualifier-const-init.c new file mode 100644 index 0000000000000..174f328628f19 --- /dev/null +++ b/clang/test/CodeGen/ptrauth-qualifier-const-init.c @@ -0,0 +1,86 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -fptrauth-calls -fptrauth-intrinsics -emit-llvm %s -o - | FileCheck %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -fptrauth-calls -fptrauth-intrinsics -emit-llvm %s -o - | FileCheck %s + +// Constant initializers for data pointers. +extern int external_int; + +// CHECK: @g1 = global ptr ptrauth (ptr @external_int, i32 1, i64 56) +int * __ptrauth(1,0,56) g1 = &external_int; + +// CHECK: @g2 = global ptr ptrauth (ptr @external_int, i32 1, i64 1272, ptr @g2) +int * __ptrauth(1,1,1272) g2 = &external_int; + +// CHECK: @g3 = global ptr null +int * __ptrauth(1,1,871) g3 = 0; + +// FIXME: should we make a ptrauth constant for this absolute symbol? +// CHECK: @g4 = global ptr inttoptr (i64 1230 to ptr) +int * __ptrauth(1,1,1902) g4 = (int*) 1230; + +// CHECK: @ga = global [3 x ptr] [ +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 712, ptr @ga), +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 712, ptr getelementptr inbounds ([3 x ptr], ptr @ga, i32 0, i32 1)), +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 712, ptr getelementptr inbounds ([3 x ptr], ptr @ga, i32 0, i32 2))] +int * __ptrauth(1,1,712) ga[3] = { &external_int, &external_int, &external_int }; + +struct A { + int * __ptrauth(1,0,431) f0; + int * __ptrauth(1,0,9182) f1; + int * __ptrauth(1,0,783) f2; +}; + +// CHECK: @gs1 = global %struct.A { +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 431), +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 9182), +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 783) } +struct A gs1 = { &external_int, &external_int, &external_int }; + +struct B { + int * __ptrauth(1,1,1276) f0; + int * __ptrauth(1,1,23674) f1; + int * __ptrauth(1,1,163) f2; +}; + +// CHECK: @gs2 = global %struct.B { +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 1276, ptr @gs2), +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 23674, ptr getelementptr inbounds (%struct.B, ptr @gs2, i32 0, i32 1)), +// CHECK-SAME: ptr ptrauth (ptr @external_int, i32 1, i64 163, ptr getelementptr inbounds (%struct.B, ptr @gs2, i32 0, i32 2)) } +struct B gs2 = { &external_int, &external_int, &external_int }; + +// Constant initializers for function pointers. +extern void external_function(void); +typedef void (*fpt)(void); + +// CHECK: @f1 = global ptr ptrauth (ptr @external_function, i32 1, i64 56) +fpt __ptrauth(1,0,56) f1 = &external_function; + +// CHECK: @f2 = global ptr ptrauth (ptr @external_function, i32 1, i64 1272, ptr @f2) +fpt __ptrauth(1,1,1272) f2 = &external_function; + +// CHECK: @fa = global [3 x ptr] [ +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 712, ptr @fa), +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 712, ptr getelementptr inbounds ([3 x ptr], ptr @fa, i32 0, i32 1)), +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 712, ptr getelementptr inbounds ([3 x ptr], ptr @fa, i32 0, i32 2))] +fpt __ptrauth(1,1,712) fa[3] = { &external_function, &external_function, &external_function }; + +struct C { + fpt __ptrauth(1,0,431) f0; + fpt __ptrauth(1,0,9182) f1; + fpt __ptrauth(1,0,783) f2; +}; +// CHECK: @fs1 = global %struct.C { +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 431), +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 9182), +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 783) } +struct C fs1 = { &external_function, &external_function, &external_function }; + +struct D { + fpt __ptrauth(1,1,1276) f0; + fpt __ptrauth(1,1,23674) f1; + fpt __ptrauth(1,1,163) f2; +}; +// CHECK: @fs2 = global %struct.D { +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 1276, ptr @fs2), +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 23674, ptr getelementptr inbounds (%struct.D, ptr @fs2, i32 0, i32 1)), +// CHECK-SAME: ptr ptrauth (ptr @external_function, i32 1, i64 163, ptr getelementptr inbounds (%struct.D, ptr @fs2, i32 0, i32 2)) } +struct D fs2 = { &external_function, &external_function, &external_function }; diff --git a/clang/test/CodeGen/ptrauth-qualifier-function.c b/clang/test/CodeGen/ptrauth-qualifier-function.c new file mode 100644 index 0000000000000..cd25b77a01548 --- /dev/null +++ b/clang/test/CodeGen/ptrauth-qualifier-function.c @@ -0,0 +1,145 @@ +// RUN: %clang_cc1 %s -fptrauth-function-pointer-type-discrimination -triple arm64-apple-ios13 -fptrauth-calls -fptrauth-intrinsics -disable-llvm-passes -emit-llvm -o- | FileCheck --check-prefixes=CHECK,TYPE %s +// RUN: %clang_cc1 %s -fptrauth-function-pointer-type-discrimination -triple aarch64-linux-gnu -fptrauth-calls -fptrauth-intrinsics -disable-llvm-passes -emit-llvm -o- | FileCheck --check-prefixes=CHECK,TYPE %s +// RUN: %clang_cc1 %s -triple arm64-apple-ios13 -fptrauth-calls -fptrauth-intrinsics -disable-llvm-passes -emit-llvm -o- | FileCheck --check-prefixes=CHECK,ZERO %s +// RUN: %clang_cc1 %s -triple aarch64-linux-gnu -fptrauth-calls -fptrauth-intrinsics -disable-llvm-passes -emit-llvm -o- | FileCheck --check-prefixes=CHECK,ZERO %s +// RUN: %clang_cc1 -xc++ %s -fptrauth-function-pointer-type-discrimination -triple arm64-apple-ios13 -fptrauth-calls -fptrauth-intrinsics -disable-llvm-passes -emit-llvm -o- | FileCheck --check-prefixes=CHECK,TYPE,CHECK-CXX %s +// RUN: %clang_cc1 -xc++ %s -fptrauth-function-pointer-type-discrimination -triple aarch64-linux-gnu -fptrauth-calls -fptrauth-intrinsics -disable-llvm-passes -emit-llvm -o- | FileCheck --check-prefixes=CHECK,TYPE,CHECK-CXX %s + +#ifdef __cplusplus +extern "C" { +#endif + +void (*fptr)(void); +void (* __ptrauth(0, 0, 42) f2ptr_42_discm)(int); +void f(int); +void (* const __ptrauth(0, 0, 42) f_const_ptr)(int) = &f; + +// CHECK-LABEL: define {{.*}}void @test_assign_to_qualified +void test_assign_to_qualified() { + f2ptr_42_discm = (void (*)(int))fptr; + + // CHECK: [[ENTRY:.*]]:{{$}} + // CHECK: [[FPTR:%.*]] = load ptr, ptr @fptr + // CHECK-NEXT: [[CMP:%.*]] = icmp ne ptr [[FPTR]], null + // TYPE-NEXT: br i1 [[CMP]], label %[[RESIGN1:.*]], label %[[JOIN1:.*]] + // ZERO-NEXT: br i1 [[CMP]], label %[[RESIGN2:.*]], label %[[JOIN2:.*]] + + // TYPE: [[RESIGN1]]: + // TYPE-NEXT: [[FPTR2:%.*]] = ptrtoint ptr [[FPTR]] to i64 + // TYPE-NEXT: [[FPTR4:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[FPTR2]], i32 0, i64 18983, i32 0, i64 2712) + // TYPE-NEXT: [[FPTR5:%.*]] = inttoptr i64 [[FPTR4]] to ptr + // TYPE-NEXT: br label %[[JOIN1]] + + // TYPE: [[JOIN1]]: + // TYPE-NEXT: [[FPTR6:%.*]] = phi ptr [ null, %[[ENTRY]] ], [ [[FPTR5]], %[[RESIGN1]] ] + // TYPE-NEXT: [[CMP:%.*]] = icmp ne ptr [[FPTR6]], null + // TYPE-NEXT: br i1 [[CMP]], label %[[RESIGN2:.*]], label %[[JOIN2:.*]] + + // CHECK: [[RESIGN2]]: + // TYPE-NEXT: [[FPTR7:%.*]] = ptrtoint ptr [[FPTR6]] to i64 + // TYPE-NEXT: [[FPTR8:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[FPTR7]], i32 0, i64 2712, i32 0, i64 42) + // ZERO-NEXT: [[FPTR7:%.*]] = ptrtoint ptr [[FPTR]] to i64 + // ZERO-NEXT: [[FPTR8:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[FPTR7]], i32 0, i64 0, i32 0, i64 42) + // CHECK-NEXT: [[FPTR9:%.*]] = inttoptr i64 [[FPTR8]] to ptr + // CHECK-NEXT: br label %[[JOIN2]] + + // CHECK: [[JOIN2]] + // TYPE-NEXT: [[FPTR10:%.*]] = phi ptr [ null, %[[JOIN1]] ], [ [[FPTR9]], %[[RESIGN2]] ] + // ZERO-NEXT: [[FPTR10:%.*]] = phi ptr [ null, %[[ENTRY]] ], [ [[FPTR9]], %[[RESIGN2]] ] + // CHECK-NEXT store void (i32)* [[FPTR10]], void (i32)** @f2ptr_42_discm +} + +// CHECK-LABEL: define {{.*}}void @test_assign_from_qualified +void test_assign_from_qualified() { + fptr = (void (*)(void))f2ptr_42_discm; + + // CHECK: [[ENTRY:.*]]:{{$}} + // CHECK: [[FPTR:%.*]] = load ptr, ptr @f2ptr_42_discm + // CHECK-NEXT: [[CMP:%.*]] = icmp ne ptr [[FPTR]], null + // TYPE-NEXT: br i1 [[CMP]], label %[[RESIGN1:.*]], label %[[JOIN1:.*]] + // ZERO-NEXT: br i1 [[CMP]], label %[[RESIGN2:.*]], label %[[JOIN2:.*]] + + // TYPE: [[RESIGN1]]: + // TYPE-NEXT: [[FPTR1:%.*]] = ptrtoint ptr [[FPTR]] to i64 + // TYPE-NEXT: [[FPTR2:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[FPTR1]], i32 0, i64 42, i32 0, i64 2712) + // TYPE-NEXT: [[FPTR3:%.*]] = inttoptr i64 [[FPTR2]] to ptr + // TYPE-NEXT: br label %[[JOIN1]] + + // TYPE: [[JOIN1]]: + // TYPE-NEXT: [[FPTR4:%.*]] = phi ptr [ null, %[[ENTRY]] ], [ [[FPTR3]], %[[RESIGN1]] ] + // TYPE-NEXT: [[CMP:%.*]] = icmp ne ptr [[FPTR4]], null + // TYPE-NEXT: br i1 [[CMP]], label %[[RESIGN2:.*]], label %[[JOIN2:.*]] + + // CHECK: [[RESIGN2]]: + // TYPE-NEXT: [[FPTR6:%.*]] = ptrtoint ptr [[FPTR4]] to i64 + // TYPE-NEXT: [[FPTR7:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[FPTR6]], i32 0, i64 2712, i32 0, i64 18983) + // ZERO-NEXT: [[FPTR6:%.*]] = ptrtoint ptr [[FPTR]] to i64 + // ZERO-NEXT: [[FPTR7:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[FPTR6]], i32 0, i64 42, i32 0, i64 0) + // CHECK-NEXT: [[FPTR8:%.*]] = inttoptr i64 [[FPTR7]] to ptr + // CHECK-NEXT: br label %[[JOIN2]] + + // CHECK: [[JOIN2]] + // TYPE-NEXT: [[FPTR9:%.*]] = phi ptr [ null, %[[JOIN1]] ], [ [[FPTR8]], %[[RESIGN2]] ] + // ZERO-NEXT: [[FPTR9:%.*]] = phi ptr [ null, %[[ENTRY]] ], [ [[FPTR8]], %[[RESIGN2]] ] + // CHECK-NEXT store void ()* [[FPTR10]], void ()** @f2ptr_42_discm +} + +// CHECK-LABEL: define {{.*}}void @test_const_ptr_function_call() +void test_const_ptr_function_call(void) { + f_const_ptr(1); + + // TYPE: call void ptrauth (ptr @f, i32 0, i64 2712)(i32 noundef 1) [ "ptrauth"(i32 0, i64 2712) ] + // ZERO: call void ptrauth (ptr @f, i32 0)(i32 noundef 1) [ "ptrauth"(i32 0, i64 0) ] +} + +#ifdef __cplusplus +void (* get_fptr(void))(int); +void (* __ptrauth(0, 0, 42) f_const_ptr2)(int) = get_fptr(); +void (* const __ptrauth(0, 1, 43) &f_ref)(int) = f_const_ptr2; + +// CHECK-CXX-LABEL: define {{.*}}internal void @__cxx_global_var_init() +// CHECK-CXX: [[ENTRY:.*]]: +// CHECK-CXX: %[[CALL:.*]] = call ptr @get_fptr() +// CHECK-CXX: %[[V0:.*]] = icmp ne ptr %[[CALL]], null +// CHECK-CXX: br i1 %[[V0]], label %[[RESIGN_NONNULL:.*]], label %[[RESIGN_CONT:.*]] + +// CHECK-CXX: [[RESIGN_NONNULL]]: +// CHECK-CXX: %[[V1:.*]] = ptrtoint ptr %[[CALL]] to i64 +// CHECK-CXX: %[[V2:.*]] = call i64 @llvm.ptrauth.resign(i64 %[[V1]], i32 0, i64 2712, i32 0, i64 42) +// CHECK-CXX: %[[V3:.*]] = inttoptr i64 %[[V2]] to ptr +// CHECK-CXX: br label %[[RESIGN_CONT]] + +// CHECK-CXX: [[RESIGN_CONT]]: +// CHECK-CXX: %[[V4:.*]] = phi ptr [ null, %[[ENTRY]] ], [ %[[V3]], %[[RESIGN_NONNULL]] ] +// CHECK-CXX: store ptr %[[V4]], ptr @f_const_ptr2, align 8 + +// CHECK-CXX-LABEL: define {{.*}}internal void @__cxx_global_var_init.1() +// CHECK-CXX: [[ENTRY:.*]]: +// CHECK-CXX: %[[V0:.*]] = load ptr, ptr @f_const_ptr2, align 8 +// CHECK-CXX: %[[V1:.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @_ZGR5f_ref_ to i64), i64 43) +// CHECK-CXX: %[[V2:.*]] = icmp ne ptr %[[V0]], null +// CHECK-CXX: br i1 %[[V2]], label %[[RESIGN_NONNULL:.*]], label %[[RESIGN_CONT:.*]] + +// CHECK-CXX: [[RESIGN_NONNULL]]: +// CHECK-CXX: %[[V3:.*]] = ptrtoint ptr %[[V0]] to i64 +// CHECK-CXX: %[[V4:.*]] = call i64 @llvm.ptrauth.resign(i64 %[[V3]], i32 0, i64 42, i32 0, i64 %[[V1]]) +// CHECK-CXX: %[[V5:.*]] = inttoptr i64 %[[V4]] to ptr +// CHECK-CXX: br label %[[RESIGN_CONT]] + +// CHECK-CXX: [[RESIGN_CONT]]: +// CHECK-CXX: %[[V6:.*]] = phi ptr [ null, %[[ENTRY]] ], [ %[[V5]], %[[RESIGN_NONNULL]] ] +// CHECK-CXX: store ptr %[[V6]], ptr @_ZGR5f_ref_, align 8 +// CHECK-CXX: store ptr @_ZGR5f_ref_, ptr @f_ref, align 8 + +// CHECK-CXX-LABEL: define {{.*}}void @test_const_ptr_ref_function_call() +void test_const_ptr_ref_function_call(void) { + f_ref(1); + + // CHECK-CXX: %[[V0:.*]] = load ptr, ptr @f_ref, align 8 + // CHECK-CXX: %[[V1:.*]] = load ptr, ptr %[[V0]], align 8 + // CHECK-CXX: %[[V2:.*]] = ptrtoint ptr %[[V0]] to i64 + // CHECK-CXX: %[[V3:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V2]], i64 43) + // CHECK-CXX: call void %[[V1]](i32 noundef 1) [ "ptrauth"(i32 0, i64 %[[V3]]) ] +} +} +#endif diff --git a/clang/test/CodeGen/ptrauth-qualifier-loadstore.c b/clang/test/CodeGen/ptrauth-qualifier-loadstore.c new file mode 100644 index 0000000000000..db259ed950fec --- /dev/null +++ b/clang/test/CodeGen/ptrauth-qualifier-loadstore.c @@ -0,0 +1,745 @@ +// RUN: %clang_cc1 -fptrauth-function-pointer-type-discrimination -triple arm64-apple-ios -fptrauth-calls -fptrauth-intrinsics -emit-llvm %s -o - | FileCheck %s +// RUN: %clang_cc1 -fptrauth-function-pointer-type-discrimination -triple aarch64-linux-gnu -fptrauth-calls -fptrauth-intrinsics -emit-llvm %s -o - | FileCheck %s + +#define IQ __ptrauth(1,0,50) +#define AQ __ptrauth(1,1,50) +#define DIFF_IQ __ptrauth(1,0,100) +#define DIFF_AQ __ptrauth(1,1,100) +#define ZERO_IQ __ptrauth(1,0,0) +#define ZERO_AQ __ptrauth(1,1,0) + +extern int external_int; +extern int * global_upi; +extern int * IQ global_iqpi; +extern int * AQ global_aqpi; +extern void use_upi(int *ptr); + +typedef void func_t(void); +extern void external_func(void); +extern func_t *global_upf; +extern func_t * IQ global_iqpf; +extern func_t * AQ global_aqpf; +extern void use_upf(func_t *ptr); + +// Data with address-independent qualifiers. + +// CHECK-LABEL: define {{.*}}void @test_store_data_i_constant() +void test_store_data_i_constant() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[SIGN:%.*]] = call i64 @llvm.ptrauth.sign(i64 ptrtoint (ptr @external_int to i64), i32 1, i64 50) +// CHECK-NEXT: [[T0:%.*]] = inttoptr i64 [[SIGN]] to ptr +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * IQ iqpi = &external_int; +// CHECK-NEXT: [[T0:%.*]] = call i64 @llvm.ptrauth.sign(i64 ptrtoint (ptr @external_int to i64), i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T0]] to ptr +// CHECK-NEXT: store ptr [[SIGNED]], ptr [[V]], +// CHECK-NEXT: ret void + iqpi = &external_int; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_iu() +void test_store_data_iu() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.sign(i64 [[T0]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * IQ iqpi = global_upi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.sign(i64 [[T0]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + iqpi = global_upi; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_ia() +void test_store_data_ia() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * IQ iqpi = global_aqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + iqpi = global_aqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[RESULT:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[RESULT]], ptr [[V]], +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[RESULT]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[RESULT]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.auth(i64 [[T0]], i32 1, i64 50) +// CHECK-NEXT: [[AUTHED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[RESULT:%.*]] = phi ptr [ null, {{.*}} ], [ [[AUTHED]], {{.*}} ] +// CHECK-NEXT: call void @use_upi(ptr noundef [[RESULT]]) + use_upi(iqpi = global_aqpi); +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_ii_same() +void test_store_data_ii_same() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: store ptr [[LOAD]], ptr [[V]], + int * IQ iqpi = global_iqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: store ptr [[LOAD]], ptr [[V]], + iqpi = global_iqpi; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_ii_different() +void test_store_data_ii_different() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 100) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * DIFF_IQ iqpi = global_iqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 100) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + iqpi = global_iqpi; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_ii_zero() +void test_store_data_ii_zero() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 0) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * ZERO_IQ iqpi = global_iqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr [[V]] +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 0, i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr @global_iqpi, + global_iqpi = iqpi; +} + +// CHECK-LABEL: define {{.*}}void @test_load_data_i() +void test_load_data_i() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.auth(i64 [[T0]], i32 1, i64 50) +// CHECK-NEXT: [[AUTHED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[AUTHED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int *upi = global_iqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.auth(i64 [[T0]], i32 1, i64 50) +// CHECK-NEXT: [[AUTHED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[AUTHED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + upi = global_iqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.auth(i64 [[T0]], i32 1, i64 50) +// CHECK-NEXT: [[AUTHED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[AUTHED]], {{.*}} ] +// CHECK-NEXT: call void @use_upi(ptr noundef [[T0]]) + use_upi(global_iqpi); +} + +// Data with address-discriminated qualifiers. + +// CHECK-LABEL: define {{.*}}void @test_store_data_a_constant() +void test_store_data_a_constant() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[SIGN:%.*]] = call i64 @llvm.ptrauth.sign(i64 ptrtoint (ptr @external_int to i64), i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[T0:%.*]] = inttoptr i64 [[SIGN]] to ptr +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * AQ aqpi = &external_int; +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[SIGN:%.*]] = call i64 @llvm.ptrauth.sign(i64 ptrtoint (ptr @external_int to i64), i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[T0:%.*]] = inttoptr i64 [[SIGN]] to ptr +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpi = &external_int; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_au() +void test_store_data_au() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upi, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.sign(i64 [[T0]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * AQ aqpi = global_upi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upi, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.sign(i64 [[T0]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpi = global_upi; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_ai() +void test_store_data_ai() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * AQ aqpi = global_iqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpi, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpi = global_iqpi; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_aa_same() +void test_store_data_aa_same() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * AQ aqpi = global_aqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpi = global_aqpi; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_aa_different() +void test_store_data_aa_different() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 100) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * DIFF_AQ aqpi = global_aqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 100) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpi = global_aqpi; +} + +// CHECK-LABEL: define {{.*}}void @test_store_data_aa_zero() +void test_store_data_aa_zero() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[NEWDISC:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int * ZERO_AQ aqpi = global_aqpi; +// CHECK: [[LOAD:%.*]] = load ptr, ptr [[V]], +// CHECK-NEXT: [[OLDDISC:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr @global_aqpi, + global_aqpi = aqpi; +} + +// CHECK-LABEL: define {{.*}}void @test_load_data_a() +void test_load_data_a() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.auth(i64 [[T0]], i32 1, i64 [[OLDDISC]]) +// CHECK-NEXT: [[AUTHED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[AUTHED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + int *upi = global_aqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.auth(i64 [[T0]], i32 1, i64 [[OLDDISC]]) +// CHECK-NEXT: [[AUTHED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[AUTHED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + upi = global_aqpi; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpi, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpi to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.auth(i64 [[T0]], i32 1, i64 [[OLDDISC]]) +// CHECK-NEXT: [[AUTHED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[AUTHED]], {{.*}} ] +// CHECK-NEXT: call void @use_upi(ptr noundef [[T0]]) + use_upi(global_aqpi); +} + +// Function with address-independent qualifiers. + +// CHECK-LABEL: define {{.*}}void @test_store_function_i_constant() +void test_store_function_i_constant() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[SIGN:%.*]] = call i64 @llvm.ptrauth.resign(i64 ptrtoint (ptr ptrauth (ptr @external_func, i32 0, i64 18983) to i64), i32 0, i64 18983, i32 1, i64 50) +// CHECK-NEXT: [[T0:%.*]] = inttoptr i64 [[SIGN]] to ptr +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * IQ iqpf = &external_func; +// CHECK-NEXT: [[SIGN:%.*]] = call i64 @llvm.ptrauth.resign(i64 ptrtoint (ptr ptrauth (ptr @external_func, i32 0, i64 18983) to i64), i32 0, i64 18983, i32 1, i64 50) +// CHECK-NEXT: [[T0:%.*]] = inttoptr i64 [[SIGN]] to ptr +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + iqpf = &external_func; +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_iu() +void test_store_function_iu() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upf, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 0, i64 18983, i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * IQ iqpf = global_upf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upf, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 0, i64 18983, i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + iqpf = global_upf; +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_ia() +void test_store_function_ia() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * IQ iqpf = global_aqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + iqpf = global_aqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 50) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[RESULT:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[RESULT]], ptr [[V]], +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[RESULT]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[RESULT]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 0, i64 18983) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: call void @use_upf(ptr noundef [[T0]]) + use_upf(iqpf = global_aqpf); +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_ii_same() +void test_store_function_ii_same() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: store ptr [[LOAD]], ptr [[V]], + func_t * IQ iqpf = global_iqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: store ptr [[LOAD]], ptr [[V]], + iqpf = global_iqpf; +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_ii_different() +void test_store_function_ii_different() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 100) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * DIFF_IQ iqpf = global_iqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 100) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + iqpf = global_iqpf; +} + +// CHECK-LABEL: define {{.*}}void @test_load_function_i() +void test_load_function_i() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 0, i64 18983) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t *upf = global_iqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 0, i64 18983) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + upf = global_iqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 0, i64 18983) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: call void @use_upf(ptr noundef [[T0]]) + use_upf(global_iqpf); +} + +// Function with address-discriminated qualifiers. + +// CHECK-LABEL: define {{.*}}void @test_store_function_a_constant() +void test_store_function_a_constant() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[SIGN:%.*]] = call i64 @llvm.ptrauth.resign(i64 ptrtoint (ptr ptrauth (ptr @external_func, i32 0, i64 18983) to i64), i32 0, i64 18983, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[T0:%.*]] = inttoptr i64 [[SIGN]] to ptr +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * AQ aqpf = &external_func; +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[SIGN:%.*]] = call i64 @llvm.ptrauth.resign(i64 ptrtoint (ptr ptrauth (ptr @external_func, i32 0, i64 18983) to i64), i32 0, i64 18983, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[T0:%.*]] = inttoptr i64 [[SIGN]] to ptr +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpf = &external_func; +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_au() +void test_store_function_au() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upf, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 0, i64 18983, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * AQ aqpf = global_upf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_upf, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 0, i64 18983, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpf = global_upf; +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_ai() +void test_store_function_ai() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * AQ aqpf = global_iqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_iqpf, +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 50, i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpf = global_iqpf; +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_aa_same() +void test_store_function_aa_same() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * AQ aqpf = global_aqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpf = global_aqpf; +} + +// CHECK-LABEL: define {{.*}}void @test_store_function_aa_different() +void test_store_function_aa_different() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 100) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t * DIFF_AQ aqpf = global_aqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = ptrtoint ptr [[V]] to i64 +// CHECK-NEXT: [[NEWDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 [[T0]], i64 100) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 1, i64 [[NEWDISC]]) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + aqpf = global_aqpf; +} + +// CHECK-LABEL: define {{.*}}void @test_load_function_a() +void test_load_function_a() { +// CHECK: [[V:%.*]] = alloca ptr, +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 0, i64 18983) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + func_t *upf = global_aqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 0, i64 18983) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: store ptr [[T0]], ptr [[V]], + upf = global_aqpf; +// CHECK-NEXT: [[LOAD:%.*]] = load ptr, ptr @global_aqpf, +// CHECK-NEXT: [[OLDDISC:%.*]] = call i64 @llvm.ptrauth.blend(i64 ptrtoint (ptr @global_aqpf to i64), i64 50) +// CHECK-NEXT: [[T0:%.*]] = icmp ne ptr [[LOAD]], null +// CHECK-NEXT: br i1 [[T0]], +// CHECK: [[T0:%.*]] = ptrtoint ptr [[LOAD]] to i64 +// CHECK-NEXT: [[T1:%.*]] = call i64 @llvm.ptrauth.resign(i64 [[T0]], i32 1, i64 [[OLDDISC]], i32 0, i64 18983) +// CHECK-NEXT: [[SIGNED:%.*]] = inttoptr i64 [[T1]] to ptr +// CHECK-NEXT: br label +// CHECK: [[T0:%.*]] = phi ptr [ null, {{.*}} ], [ [[SIGNED]], {{.*}} ] +// CHECK-NEXT: call void @use_upf(ptr noundef [[T0]]) + use_upf(global_aqpf); +} diff --git a/clang/test/CodeGenCXX/mangle-itanium-ptrauth.cpp b/clang/test/CodeGenCXX/mangle-itanium-ptrauth.cpp new file mode 100644 index 0000000000000..88d80423c3764 --- /dev/null +++ b/clang/test/CodeGenCXX/mangle-itanium-ptrauth.cpp @@ -0,0 +1,12 @@ +// RUN: %clang_cc1 -std=c++11 -fptrauth-intrinsics -fptrauth-calls -emit-llvm -o - -triple=arm64-apple-ios %s | FileCheck %s +// RUN: %clang_cc1 -std=c++11 -fptrauth-intrinsics -fptrauth-calls -emit-llvm -o - -triple=aarch64-linux-gnu %s | FileCheck %s + +// CHECK: define {{.*}}void @_Z3fooPU9__ptrauthILj3ELb1ELj234EEPi( +void foo(int * __ptrauth(3, 1, 234) *) {} + +template +void foo(T t) {} + +// CHECK: define weak_odr void @_Z3fooIPU9__ptrauthILj1ELb0ELj64EEPiEvT_( +template void foo(int * __ptrauth(1, 0, 64) *); + diff --git a/clang/test/CodeGenCXX/mangle-ms-ptrauth.cpp b/clang/test/CodeGenCXX/mangle-ms-ptrauth.cpp new file mode 100644 index 0000000000000..95e5efa472dfd --- /dev/null +++ b/clang/test/CodeGenCXX/mangle-ms-ptrauth.cpp @@ -0,0 +1,17 @@ +// RUN: %clang_cc1 -std=c++11 -fptrauth-intrinsics -fptrauth-calls -emit-llvm -o - -triple=aarch64-windows-msvc %s | FileCheck %s + +template +struct S {}; + +// CHECK: @"?s@@3U?$S@PE__ptrauth1A@ENC@AH@@A" = +S s; + +// CHECK: define dso_local void @"?foo@@YAXPEAPE__ptrauth20OK@AH@Z"( +void foo(int * __ptrauth(3, 1, 234) *) {} + +template +void foo(T t) {} + +// CHECK: define weak_odr dso_local void @"??$foo@PEAPE__ptrauth0A@EA@AH@@YAXPEAPE__ptrauth0A@EA@AH@Z"( +template void foo(int * __ptrauth(1, 0, 64) *); + diff --git a/clang/test/CodeGenCXX/ptrauth-qualifier-struct.cpp b/clang/test/CodeGenCXX/ptrauth-qualifier-struct.cpp new file mode 100644 index 0000000000000..7d6de50d926b5 --- /dev/null +++ b/clang/test/CodeGenCXX/ptrauth-qualifier-struct.cpp @@ -0,0 +1,168 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -fptrauth-calls -fptrauth-intrinsics -std=c++11 -emit-llvm %s -o - | FileCheck -check-prefixes=CHECK,IOS %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -fptrauth-calls -fptrauth-intrinsics -std=c++11 -emit-llvm %s -o - | FileCheck %s + +#define AQ __ptrauth(1,1,50) +#define IQ __ptrauth(1,0,50) + +// CHECK: %[[STRUCT_SA:.*]] = type { ptr, ptr } +// CHECK: %[[STRUCT_SI:.*]] = type { ptr } + +struct SA { + int * AQ m0; // Signed using address discrimination. + int * AQ m1; // Signed using address discrimination. +}; + +struct SI { + int * IQ m; // No address discrimination. +}; + +struct __attribute__((trivial_abi)) TrivialSA { + int * AQ m0; // Signed using address discrimination. + int * AQ m1; // Signed using address discrimination. +}; + +// Check that TrivialSA is passed indirectly despite being annotated with +// 'trivial_abi'. + +// CHECK: define {{.*}}void @_Z18testParamTrivialSA9TrivialSA(ptr noundef %{{.*}}) + +void testParamTrivialSA(TrivialSA a) { +} + +// CHECK: define {{.*}}void @_Z19testCopyConstructor2SA(ptr +// CHECK: call {{.*}}@_ZN2SAC1ERKS_( + +// CHECK: define linkonce_odr {{.*}}@_ZN2SAC1ERKS_( +// CHECK: call {{.*}}@_ZN2SAC2ERKS_( + +void testCopyConstructor(SA a) { + SA t = a; +} + +// CHECK: define {{.*}}void @_Z19testMoveConstructor2SA(ptr +// CHECK: call {{.*}}@_ZN2SAC1EOS_( + +// CHECK: define linkonce_odr {{.*}}@_ZN2SAC1EOS_( +// CHECK: call {{.*}}@_ZN2SAC2EOS_( + +void testMoveConstructor(SA a) { + SA t = static_cast(a); +} + +// CHECK: define {{.*}}void @_Z18testCopyAssignment2SA(ptr +// CHECK: call noundef nonnull align 8 dereferenceable(16) ptr @_ZN2SAaSERKS_( + +// CHECK: define {{.*}}linkonce_odr noundef nonnull align 8 dereferenceable(16) ptr @_ZN2SAaSERKS_(ptr noundef nonnull align 8 dereferenceable(16) %[[THIS:.*]], ptr noundef nonnull align 8 dereferenceable(16) %0) +// CHECK: %[[THIS_ADDR:.*]] = alloca ptr, align 8 +// CHECK: %[[_ADDR:.*]] = alloca ptr, align 8 +// CHECK: store ptr %[[THIS]], ptr %[[THIS_ADDR]], align 8 +// CHECK: store ptr %[[V0:.*]], ptr %[[_ADDR]], align 8 +// CHECK: %[[THISI:.*]] = load ptr, ptr %[[THIS_ADDR]], align 8 +// CHECK: %[[M0:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[THISI]], i32 0, i32 0 +// CHECK: %[[V1:.*]] = load ptr, ptr %[[_ADDR]], align 8 +// CHECK: %[[M02:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[V1]], i32 0, i32 0 +// CHECK: %[[V2:.*]] = load ptr, ptr %[[M02]], align 8 +// CHECK: %[[V3:.*]] = ptrtoint ptr %[[M02]] to i64 +// CHECK: %[[V4:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V3]], i64 50) +// CHECK: %[[V5:.*]] = ptrtoint ptr %[[M0]] to i64 +// CHECK: %[[V6:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V5]], i64 50) +// CHECK: %[[V8:.*]] = ptrtoint ptr %[[V2]] to i64 +// CHECK: %[[V9:.*]] = call i64 @llvm.ptrauth.resign(i64 %[[V8]], i32 1, i64 %[[V4]], i32 1, i64 %[[V6]]) + +void testCopyAssignment(SA a) { + SA t; + t = a; +} + +// CHECK: define {{.*}}void @_Z18testMoveAssignment2SA(ptr +// CHECK: call noundef nonnull align 8 dereferenceable(16) ptr @_ZN2SAaSEOS_( + +// CHECK: define {{.*}}linkonce_odr noundef nonnull align 8 dereferenceable(16) ptr @_ZN2SAaSEOS_(ptr noundef nonnull align 8 dereferenceable(16) %[[THIS:.*]], ptr noundef nonnull align 8 dereferenceable(16) %0) +// CHECK: %[[THIS_ADDR:.*]] = alloca ptr, align 8 +// CHECK: %[[_ADDR:.*]] = alloca ptr, align 8 +// CHECK: store ptr %[[THIS]], ptr %[[THIS_ADDR]], align 8 +// CHECK: store ptr %[[V0:.*]], ptr %[[_ADDR]], align 8 +// CHECK: %[[THISI:.*]] = load ptr, ptr %[[THIS_ADDR]], align 8 +// CHECK: %[[M0:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[THISI]], i32 0, i32 0 +// CHECK: %[[V1:.*]] = load ptr, ptr %[[_ADDR]], align 8 +// CHECK: %[[M02:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[V1]], i32 0, i32 0 +// CHECK: %[[V2:.*]] = load ptr, ptr %[[M02]], align 8 +// CHECK: %[[V3:.*]] = ptrtoint ptr %[[M02]] to i64 +// CHECK: %[[V4:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V3]], i64 50) +// CHECK: %[[V5:.*]] = ptrtoint ptr %[[M0]] to i64 +// CHECK: %[[V6:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V5]], i64 50) +// CHECK: %[[V8:.*]] = ptrtoint ptr %[[V2]] to i64 +// CHECK: %[[V9:.*]] = call i64 @llvm.ptrauth.resign(i64 %[[V8]], i32 1, i64 %[[V4]], i32 1, i64 %[[V6]]) + +void testMoveAssignment(SA a) { + SA t; + t = static_cast(a); +} + +// CHECK: define {{.*}}void @_Z19testCopyConstructor2SI(i +// CHECK: call void @llvm.memcpy.p0.p0.i64( + +void testCopyConstructor(SI a) { + SI t = a; +} + +// CHECK: define {{.*}}void @_Z19testMoveConstructor2SI( +// CHECK: call void @llvm.memcpy.p0.p0.i64( + +void testMoveConstructor(SI a) { + SI t = static_cast(a); +} + +// CHECK: define {{.*}}void @_Z18testCopyAssignment2SI( +// CHECK: call void @llvm.memcpy.p0.p0.i64( + +void testCopyAssignment(SI a) { + SI t; + t = a; +} + +// CHECK: define {{.*}}void @_Z18testMoveAssignment2SI( +// CHECK: call void @llvm.memcpy.p0.p0.i64( + +void testMoveAssignment(SI a) { + SI t; + t = static_cast(a); +} + +// CHECK: define linkonce_odr {{.*}}@_ZN2SAC2ERKS_(ptr noundef nonnull align 8 dereferenceable(16) %[[THIS:.*]], ptr noundef nonnull align 8 dereferenceable(16) %0) +// IOS: %[[RETVAL:.*]] = alloca ptr, align 8 +// CHECK: %[[THIS_ADDR:.*]] = alloca ptr, align 8 +// CHECK: %[[_ADDR:.*]] = alloca ptr, align 8 +// CHECK: store ptr %[[THIS]], ptr %[[THIS_ADDR]], align 8 +// CHECK: store ptr %[[V0:.*]], ptr %[[_ADDR]], align 8 +// CHECK: %[[THIS1:.*]] = load ptr, ptr %[[THIS_ADDR]], align 8 +// IOS: store ptr %[[THIS1]], ptr %[[RETVAL]], align 8 +// CHECK: %[[M0:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[THIS1]], i32 0, i32 0 +// CHECK: %[[V1:.*]] = load ptr, ptr %[[_ADDR]], align 8 +// CHECK: %[[M02:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[V1]], i32 0, i32 0 +// CHECK: %[[V2:.*]] = load ptr, ptr %[[M02]], align 8 +// CHECK: %[[V3:.*]] = ptrtoint ptr %[[M02]] to i64 +// CHECK: %[[V4:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V3]], i64 50) +// CHECK: %[[V5:.*]] = ptrtoint ptr %[[M0]] to i64 +// CHECK: %[[V6:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V5]], i64 50) +// CHECK: %[[V8:.*]] = ptrtoint ptr %[[V2]] to i64 +// CHECK: %[[V9:.*]] = call i64 @llvm.ptrauth.resign(i64 %[[V8]], i32 1, i64 %[[V4]], i32 1, i64 %[[V6]]) + +// CHECK: define linkonce_odr {{.*}}@_ZN2SAC2EOS_(ptr noundef nonnull align 8 dereferenceable(16) %[[THIS:.*]], ptr noundef nonnull align 8 dereferenceable(16) %0) +// IOS: %[[RETVAL:.*]] = alloca ptr, align 8 +// CHECK: %[[THIS_ADDR:.*]] = alloca ptr, align 8 +// CHECK: %[[_ADDR:.*]] = alloca ptr, align 8 +// CHECK: store ptr %[[THIS]], ptr %[[THIS_ADDR]], align 8 +// CHECK: store ptr %[[V0:.*]], ptr %[[_ADDR]], align 8 +// CHECK: %[[THIS1:.*]] = load ptr, ptr %[[THIS_ADDR]], align 8 +// IOS: store ptr %[[THIS1]], ptr %[[RETVAL]], align 8 +// CHECK: %[[M0:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[THIS1]], i32 0, i32 0 +// CHECK: %[[V1:.*]] = load ptr, ptr %[[_ADDR]], align 8 +// CHECK: %[[M02:.*]] = getelementptr inbounds nuw %[[STRUCT_SA]], ptr %[[V1]], i32 0, i32 0 +// CHECK: %[[V2:.*]] = load ptr, ptr %[[M02]], align 8 +// CHECK: %[[V3:.*]] = ptrtoint ptr %[[M02]] to i64 +// CHECK: %[[V4:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V3]], i64 50) +// CHECK: %[[V5:.*]] = ptrtoint ptr %[[M0]] to i64 +// CHECK: %[[V6:.*]] = call i64 @llvm.ptrauth.blend(i64 %[[V5]], i64 50) +// CHECK: %[[V8:.*]] = ptrtoint ptr %[[V2]] to i64 +// CHECK: %[[V9:.*]] = call i64 @llvm.ptrauth.resign(i64 %[[V8]], i32 1, i64 %[[V4]], i32 1, i64 %[[V6]]) diff --git a/clang/test/CodeGenObjCXX/ptrauth-struct-cxx-abi.mm b/clang/test/CodeGenObjCXX/ptrauth-struct-cxx-abi.mm new file mode 100644 index 0000000000000..e5cb71bad47c0 --- /dev/null +++ b/clang/test/CodeGenObjCXX/ptrauth-struct-cxx-abi.mm @@ -0,0 +1,35 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios11 -fptrauth-calls -fptrauth-intrinsics -std=c++11 -fobjc-arc -emit-llvm -o - %s | FileCheck %s + +#define AQ __ptrauth(1,1,50) + +struct AddrDiscStrong0 { + int * AQ f0; // Signed using address discrimination. + __strong id f1; +}; + +struct AddrDiscStrong1 { + AddrDiscStrong1(const AddrDiscStrong1 &); + int * AQ f0; // Signed using address discrimination. + __strong id f1; +}; + +// Check that AddrDiscStrong0 is destructed in the callee. + +// CHECK: define void @_Z24testParamAddrDiscStrong015AddrDiscStrong0(ptr noundef %[[A:.*]]) +// CHECK: call noundef ptr @_ZN15AddrDiscStrong0D1Ev(ptr noundef nonnull align {{[0-9]+}} dereferenceable(16) %[[A]]) +// CHECK: ret void + +// CHECK: define linkonce_odr noundef ptr @_ZN15AddrDiscStrong0D1Ev( + +void testParamAddrDiscStrong0(AddrDiscStrong0 a) { +} + +// Check that AddrDiscStrong1 is not destructed in the callee because it has a +// non-trivial copy constructor. + +// CHECK: define void @_Z24testParamAddrDiscStrong115AddrDiscStrong1(ptr noundef %{{.*}}) +// CHECK-NOT: call +// CHECK: ret void + +void testParamAddrDiscStrong1(AddrDiscStrong1 a) { +} diff --git a/clang/test/Parser/ptrauth-qualifier.c b/clang/test/Parser/ptrauth-qualifier.c new file mode 100644 index 0000000000000..2071ac6c2d661 --- /dev/null +++ b/clang/test/Parser/ptrauth-qualifier.c @@ -0,0 +1,18 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -fsyntax-only -verify -fptrauth-intrinsics %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -fsyntax-only -verify -fptrauth-intrinsics %s + +#if __aarch64__ +#define VALID_DATA_KEY 2 +#else +#error Provide these constants if you port this test +#endif + +int * __ptrauth(VALID_DATA_KEY) valid0; + +typedef int *intp; + +int nonConstantGlobal = 5; + +__ptrauth int invalid0; // expected-error{{expected '('}} +__ptrauth() int invalid1; // expected-error{{expected expression}} +int * __ptrauth(VALID_DATA_KEY, 1, 1000, 12) invalid12; // expected-error{{qualifier must take between 1 and 3 arguments}} diff --git a/clang/test/Preprocessor/ptrauth_extension.c b/clang/test/Preprocessor/ptrauth_extension.c new file mode 100644 index 0000000000000..d6b79187ba62d --- /dev/null +++ b/clang/test/Preprocessor/ptrauth_extension.c @@ -0,0 +1,13 @@ +// RUN: %clang_cc1 -E %s -triple=aarch64 -fptrauth-intrinsics | \ +// RUN: FileCheck %s --check-prefixes=INTRIN + +// RUN: %clang_cc1 -E %s -triple=aarch64 -fptrauth-calls | \ +// RUN: FileCheck %s --check-prefixes=NOINTRIN + +#if __has_extension(ptrauth_qualifier) +// INTRIN: has_ptrauth_qualifier +void has_ptrauth_qualifier() {} +#else +// NOINTRIN: no_ptrauth_qualifier +void no_ptrauth_qualifier() {} +#endif diff --git a/clang/test/Sema/ptrauth-atomic-ops.c b/clang/test/Sema/ptrauth-atomic-ops.c new file mode 100644 index 0000000000000..ccb9a1abcc14d --- /dev/null +++ b/clang/test/Sema/ptrauth-atomic-ops.c @@ -0,0 +1,118 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -fsyntax-only -verify -fptrauth-intrinsics %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -fsyntax-only -verify -fptrauth-intrinsics %s + +#include + +int i; +int *__ptrauth(2, 1, 100) authenticated_ptr = &i; +int *__ptrauth(2, 0, 200) non_addr_discriminatedauthenticated_ptr = &i; +int * wat = &i; +#define ATOMIZE(p) (__typeof__(p) volatile _Atomic *)(long)(&p) + +void f() { + static int j = 1; + __c11_atomic_init(ATOMIZE(authenticated_ptr), 5); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __c11_atomic_store(ATOMIZE(authenticated_ptr), 0, memory_order_relaxed); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __c11_atomic_load(ATOMIZE(authenticated_ptr), memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __c11_atomic_store(ATOMIZE(authenticated_ptr), 1, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_store_n(ATOMIZE(authenticated_ptr), 4, memory_order_release); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_store(ATOMIZE(authenticated_ptr), j, memory_order_release); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __c11_atomic_exchange(ATOMIZE(authenticated_ptr), 1, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_exchange(ATOMIZE(authenticated_ptr), &j, &j, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __c11_atomic_fetch_add(ATOMIZE(authenticated_ptr), 1, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_fetch_add(ATOMIZE(authenticated_ptr), 3, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_fetch_sub(ATOMIZE(authenticated_ptr), 3, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_fetch_min(ATOMIZE(authenticated_ptr), 3, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_fetch_max(ATOMIZE(authenticated_ptr), 3, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __c11_atomic_fetch_and(ATOMIZE(authenticated_ptr), 1, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_fetch_and(ATOMIZE(authenticated_ptr), 3, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_fetch_or(ATOMIZE(authenticated_ptr), 3, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + __atomic_fetch_xor(ATOMIZE(authenticated_ptr), 3, memory_order_seq_cst); + // expected-error@-1 {{address argument to atomic operation must be a pointer to a non address discriminated type ('volatile __ptrauth(2,1,100) _Atomic(int *) *' invalid)}} + + __c11_atomic_init(ATOMIZE(non_addr_discriminatedauthenticated_ptr), &j); + __c11_atomic_store(ATOMIZE(non_addr_discriminatedauthenticated_ptr), 0, memory_order_relaxed); + __c11_atomic_load(ATOMIZE(non_addr_discriminatedauthenticated_ptr), memory_order_seq_cst); + __atomic_store(&j, ATOMIZE(non_addr_discriminatedauthenticated_ptr), memory_order_release); + // expected-warning@-1 {{incompatible pointer types passing 'volatile __ptrauth(2,0,200) _Atomic(int *) *' to parameter of type 'int *'}} + __c11_atomic_exchange(ATOMIZE(j), ATOMIZE(non_addr_discriminatedauthenticated_ptr), memory_order_seq_cst); + // expected-error@-1 {{incompatible pointer to integer conversion passing 'volatile __ptrauth(2,0,200) _Atomic(int *) *' to parameter of type 'typeof (j)' (aka 'int')}} + __c11_atomic_fetch_add(ATOMIZE(non_addr_discriminatedauthenticated_ptr), ATOMIZE(j), memory_order_seq_cst); + // expected-error@-1 {{incompatible pointer to integer conversion passing 'volatile _Atomic(typeof (j)) *' to parameter of type 'long'}} + __c11_atomic_fetch_and(ATOMIZE(j), ATOMIZE(non_addr_discriminatedauthenticated_ptr), memory_order_seq_cst); + // expected-error@-1 {{incompatible pointer to integer conversion passing 'volatile __ptrauth(2,0,200) _Atomic(int *) *' to parameter of type 'typeof (j)' (aka 'int')}} + + + __sync_fetch_and_add(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_fetch_and_sub(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_fetch_and_or(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_fetch_and_and(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_fetch_and_xor(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_fetch_and_nand(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + + __sync_add_and_fetch(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_sub_and_fetch(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_or_and_fetch(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_and_and_fetch(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_xor_and_fetch(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_nand_and_fetch(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + + __sync_bool_compare_and_swap(&authenticated_ptr, 1, 0); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_val_compare_and_swap(&authenticated_ptr, 1, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + + __sync_lock_test_and_set(&authenticated_ptr, 1); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + __sync_lock_release(&authenticated_ptr); + // expected-error@-1 {{address argument to __sync operation must be a pointer to a non address discriminated type ('int *__ptrauth(2,1,100)' invalid)}} + + +int i = 0; + + __sync_fetch_and_add(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_fetch_and_sub(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_fetch_and_or(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_fetch_and_and(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_fetch_and_xor(&non_addr_discriminatedauthenticated_ptr, &i); + + __sync_add_and_fetch(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_sub_and_fetch(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_or_and_fetch(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_and_and_fetch(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_xor_and_fetch(&non_addr_discriminatedauthenticated_ptr, &i); + + __sync_bool_compare_and_swap(&non_addr_discriminatedauthenticated_ptr, &i, &i); + __sync_val_compare_and_swap(&non_addr_discriminatedauthenticated_ptr, &i, &i); + + __sync_lock_test_and_set(&non_addr_discriminatedauthenticated_ptr, &i); + __sync_lock_release(&non_addr_discriminatedauthenticated_ptr); +} diff --git a/clang/test/Sema/ptrauth-qualifier.c b/clang/test/Sema/ptrauth-qualifier.c new file mode 100644 index 0000000000000..99d16b062ca6f --- /dev/null +++ b/clang/test/Sema/ptrauth-qualifier.c @@ -0,0 +1,103 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -std=c23 -fsyntax-only -verify -fptrauth-intrinsics %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -std=c23 -fsyntax-only -verify -fptrauth-intrinsics %s + +#if __has_feature(ptrauth_qualifier) +#warning __ptrauth qualifier enabled! +// expected-warning@-1 {{__ptrauth qualifier enabled!}} +#endif + +#if __aarch64__ +#define VALID_CODE_KEY 0 +#define VALID_DATA_KEY 2 +#define INVALID_KEY 200 +#else +#error Provide these constants if you port this test +#endif + +int * __ptrauth(VALID_DATA_KEY) valid0; +int *ptr0; + +typedef int *intp; + +int nonConstantGlobal = 5; + +__ptrauth(INVALID_KEY) int invalid2; // expected-error{{200 does not identify a valid pointer authentication key for the current target}} +__ptrauth(VALID_DATA_KEY) int invalid3; // expected-error {{'__ptrauth' qualifier only applies to pointer types; 'int' is invalid}} +__ptrauth(VALID_DATA_KEY) int *invalid4; // expected-error {{'__ptrauth' qualifier only applies to pointer types; 'int' is invalid}} +int * (__ptrauth(VALID_DATA_KEY) invalid5); // expected-error{{expected identifier or '('}} expected-error{{expected ')'}} expected-note {{to match this '('}} +int *__ptrauth(VALID_DATA_KEY) __ptrauth(VALID_DATA_KEY) invalid6; // expected-error{{type 'int *__ptrauth(2,0,0)' is already __ptrauth-qualified}} +int * __ptrauth(VALID_DATA_KEY, 2) invalid7; // expected-error {{invalid address discrimination flag '2'; '__ptrauth' requires '0' or '1'}} +int * __ptrauth(VALID_DATA_KEY, -1) invalid8; // expected-error {{invalid address discrimination flag '-1'; '__ptrauth' requires '0' or '1'}} +int * __ptrauth(VALID_DATA_KEY, 1, -1) invalid9; // expected-error {{invalid extra discriminator flag '-1'; '__ptrauth' requires a value between '0' and '65535'}} +int * __ptrauth(VALID_DATA_KEY, 1, 100000) invalid10; // expected-error {{invalid extra discriminator flag '100000'; '__ptrauth' requires a value between '0' and '65535'}} +int * __ptrauth(VALID_DATA_KEY, 1, nonConstantGlobal) invalid12; // expected-error {{argument to '__ptrauth' must be an integer constant expression}} +int * __ptrauth(VALID_DATA_KEY, nonConstantGlobal, 1000) invalid13; // expected-error {{argument to '__ptrauth' must be an integer constant expression}} +int * __ptrauth(nonConstantGlobal, 1, 1000) invalid14; // expected-error{{expression is not an integer constant expression}} + +int * __ptrauth(VALID_DATA_KEY) valid0; +int * __ptrauth(VALID_DATA_KEY) *valid1; +__ptrauth(VALID_DATA_KEY) intp valid2; +__ptrauth(VALID_DATA_KEY) intp *valid3; +intp __ptrauth(VALID_DATA_KEY) valid4; +intp __ptrauth(VALID_DATA_KEY) *valid5; +int * __ptrauth(VALID_DATA_KEY, 0) valid6; +int * __ptrauth(VALID_DATA_KEY, 1) valid7; +int * __ptrauth(VALID_DATA_KEY, (_Bool) 1) valid8; +int * __ptrauth(VALID_DATA_KEY, 1, 0) valid9; +int * __ptrauth(VALID_DATA_KEY, 1, 65535) valid10; + +int * __ptrauth(VALID_DATA_KEY) array0[10]; +int (* __ptrauth(VALID_DATA_KEY) array1)[10]; + +extern intp redeclaration0; // expected-note {{previous declaration}} +extern intp __ptrauth(VALID_DATA_KEY) redeclaration0; // expected-error{{redeclaration of 'redeclaration0' with a different type: '__ptrauth(2,0,0) intp' (aka 'int *__ptrauth(2,0,0)') vs 'intp' (aka 'int *')}} + +extern intp redeclaration1; // expected-note {{previous declaration}} +extern intp __ptrauth(VALID_DATA_KEY) redeclaration1; // expected-error{{redeclaration of 'redeclaration1' with a different type: '__ptrauth(2,0,0) intp' (aka 'int *__ptrauth(2,0,0)') vs 'intp' (aka 'int *')}} + +intp __ptrauth(VALID_DATA_KEY) redeclaration2; // expected-note {{previous definition}} +intp redeclaration2 = 0; // expected-error{{redefinition of 'redeclaration2' with a different type: 'intp' (aka 'int *') vs '__ptrauth(2,0,0) intp' (aka 'int *__ptrauth(2,0,0)')}} + +intp __ptrauth(VALID_DATA_KEY) redeclaration3; // expected-note {{previous definition}} +intp redeclaration3 = 0; // expected-error{{redefinition of 'redeclaration3' with a different type: 'intp' (aka 'int *') vs '__ptrauth(2,0,0) intp' (aka 'int *__ptrauth(2,0,0)')}} + +void illegal0(intp __ptrauth(VALID_DATA_KEY)); // expected-error {{parameter type may not be qualified with '__ptrauth'; type is '__ptrauth(2,0,0) intp' (aka 'int *__ptrauth(2,0,0)')}} +intp __ptrauth(VALID_DATA_KEY) illegal1(void); // expected-error {{return type may not be qualified with '__ptrauth'; type is '__ptrauth(2,0,0) intp' (aka 'int *__ptrauth(2,0,0)')}} + +static_assert(_Generic(typeof(valid0), int * __ptrauth(VALID_DATA_KEY) : 1, int * : 0, default : 0)); +static_assert(_Generic(typeof(valid0), int * __ptrauth(VALID_CODE_KEY) : 0, default : 1)); +static_assert(_Generic(typeof_unqual(valid0), int * __ptrauth(VALID_DATA_KEY) : 0, int * : 1, default : 0)); +static_assert(_Generic(valid0, int * __ptrauth(VALID_DATA_KEY) : 0, int * : 1, default : 0)); // expected-warning {{association of type 'int *__ptrauth(2,0,0)' will never be selected}} + +static_assert(_Generic(array0, int * __ptrauth(VALID_DATA_KEY) * : 1, default : 0)); +static_assert(_Generic(*array1, int * : 1, default : 0)); + +void test_code(intp p) { + p = (intp __ptrauth(VALID_DATA_KEY)) 0; // expected-error {{cannot cast to '__ptrauth'-qualified type '__ptrauth(2,0,0) intp' (aka 'int *__ptrauth(2,0,0)')}} + + __ptrauth(VALID_DATA_KEY) intp pSpecial = p; + pSpecial = p; + intp pNormal = pSpecial; + pNormal = pSpecial; + + intp __ptrauth(VALID_DATA_KEY) *ppSpecial0 = &pSpecial; + intp __ptrauth(VALID_DATA_KEY) *ppSpecial1 = &pNormal; // expected-error {{initializing '__ptrauth(2,0,0) intp *' (aka 'int *__ptrauth(2,0,0) *') with an expression of type 'intp *' (aka 'int **') changes pointer authentication of pointee type}} + intp *ppNormal0 = &pSpecial; // expected-error {{initializing 'intp *' (aka 'int **') with an expression of type '__ptrauth(2,0,0) intp *' (aka 'int *__ptrauth(2,0,0) *') changes pointer authentication of pointee type}} + intp *ppNormal1 = &pNormal; + + intp *pp5 = (p ? &pSpecial : &pNormal); // expected-error {{'__ptrauth' qualification mismatch ('__ptrauth(2,0,0) intp *' (aka 'int *__ptrauth(2,0,0) *') and 'intp *' (aka 'int **'))}} +} + +void test_array(void) { + intp __ptrauth(VALID_DATA_KEY) pSpecialArray[10]; + intp __ptrauth(VALID_DATA_KEY) *ppSpecial0 = pSpecialArray; + intp __ptrauth(VALID_DATA_KEY) *ppSpecial1 = &pSpecialArray[0]; +} + +__attribute__((overloadable)) int overload_func(int **); +__attribute__((overloadable)) float overload_func(int * __ptrauth(VALID_DATA_KEY) *); + +static_assert(_Generic(typeof(overload_func(&ptr0)), int : 1, default : 0)); +static_assert(_Generic(typeof(overload_func(&valid0)), float : 1, default : 0)); + +void func(int array[__ptrauth(VALID_DATA_KEY) 10]); // expected-error {{'__ptrauth' qualifier only applies to pointer types; 'int[10]' is invalid}} diff --git a/clang/test/SemaCXX/ptrauth-qualifier.cpp b/clang/test/SemaCXX/ptrauth-qualifier.cpp new file mode 100644 index 0000000000000..a7dc6ae2ffe86 --- /dev/null +++ b/clang/test/SemaCXX/ptrauth-qualifier.cpp @@ -0,0 +1,213 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -std=c++20 -fptrauth-calls -fptrauth-intrinsics -verify -fsyntax-only %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -std=c++20 -fptrauth-calls -fptrauth-intrinsics -verify -fsyntax-only %s + +#define AQ __ptrauth(1,1,50) +#define AQ2 __ptrauth(1,1,51) +#define IQ __ptrauth(1,0,50) + +struct __attribute__((trivial_abi)) AddrDisc { // expected-warning {{'trivial_abi' cannot be applied to 'AddrDisc'}} expected-note {{'trivial_abi' is disallowed on 'AddrDisc' because it has an address-discriminated '__ptrauth' field}} + int * AQ m0; +}; + +struct __attribute__((trivial_abi)) NoAddrDisc { + int * IQ m0; +}; + +namespace test_union { + + union U0 { + int * AQ f0; // expected-note 4 {{'U0' is implicitly deleted because variant field 'f0' has an address-discriminated '__ptrauth' qualifier}} + + // ptrauth fields that don't have an address-discriminated qualifier don't + // delete the special functions. + int * IQ f1; + }; + + union U1 { + int * AQ f0; // expected-note 8 {{'U1' is implicitly deleted because variant field 'f0' has an address-discriminated '__ptrauth' qualifier}} + U1() = default; + ~U1() = default; + U1(const U1 &) = default; // expected-warning {{explicitly defaulted copy constructor is implicitly deleted}} expected-note 2 {{explicitly defaulted function was implicitly deleted here}} expected-note{{replace 'default'}} + U1(U1 &&) = default; // expected-warning {{explicitly defaulted move constructor is implicitly deleted}} expected-note{{replace 'default'}} + U1 & operator=(const U1 &) = default; // expected-warning {{explicitly defaulted copy assignment operator is implicitly deleted}} expected-note 2 {{explicitly defaulted function was implicitly deleted here}} expected-note{{replace 'default'}} + U1 & operator=(U1 &&) = default; // expected-warning {{explicitly defaulted move assignment operator is implicitly deleted}} expected-note{{replace 'default'}} + }; + + // It's fine if the user has explicitly defined the special functions. + union U2 { + int * AQ f0; + U2() = default; + ~U2() = default; + U2(const U2 &); + U2(U2 &&); + U2 & operator=(const U2 &); + U2 & operator=(U2 &&); + }; + + // Address-discriminated ptrauth fields in anonymous union fields delete the + // defaulted copy/move constructors/assignment operators of the containing + // class. + struct S0 { + union { + int * AQ f0; // expected-note 4 {{' is implicitly deleted because variant field 'f0' has an address-discriminated '__ptrauth' qualifier}} + char f1; + }; + }; + + struct S1 { + union { + union { + int * AQ f0; // expected-note 4 {{implicitly deleted because variant field 'f0' has an address-discriminated '__ptrauth' qualifier}} + char f1; + } u; // expected-note 4 {{'S1' is implicitly deleted because field 'u' has a deleted}} + int f2; + }; + }; + + U0 *x0; + U1 *x1; + U2 *x2; + S0 *x3; + S1 *x4; + + // No diagnostics since constructors/destructors of the unions aren't deleted by default. + void testDefaultConstructor() { + U0 u0; + U1 u1; + U2 u2; + S0 s0; + S1 s1; + } + + // No diagnostics since destructors of the unions aren't deleted by default. + void testDestructor(U0 *u0, U1 *u1, U2 *u2, S0 *s0, S1 *s1) { + delete u0; + delete u1; + delete u2; + delete s0; + delete s1; + } + + void testCopyConstructor(U0 *u0, U1 *u1, U2 *u2, S0 *s0, S1 *s1) { + U0 t0(*u0); // expected-error {{call to implicitly-deleted copy constructor}} + U1 t1(*u1); // expected-error {{call to implicitly-deleted copy constructor}} + U2 t2(*u2); + S0 t3(*s0); // expected-error {{call to implicitly-deleted copy constructor}} + S1 t4(*s1); // expected-error {{call to implicitly-deleted copy constructor}} + } + + void testCopyAssignment(U0 *u0, U1 *u1, U2 *u2, S0 *s0, S1 *s1) { + *x0 = *u0; // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + *x1 = *u1; // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + *x2 = *u2; + *x3 = *s0; // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + *x4 = *s1; // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + } + + void testMoveConstructor(U0 *u0, U1 *u1, U2 *u2, S0 *s0, S1 *s1) { + U0 t0(static_cast(*u0)); // expected-error {{call to implicitly-deleted copy constructor}} + U1 t1(static_cast(*u1)); // expected-error {{call to implicitly-deleted copy constructor}} + U2 t2(static_cast(*u2)); + S0 t3(static_cast(*s0)); // expected-error {{call to implicitly-deleted copy constructor}} + S1 t4(static_cast(*s1)); // expected-error {{call to implicitly-deleted copy constructor}} + } + + void testMoveAssignment(U0 *u0, U1 *u1, U2 *u2, S0 *s0, S1 *s1) { + *x0 = static_cast(*u0); // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + *x1 = static_cast(*u1); // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + *x2 = static_cast(*u2); + *x3 = static_cast(*s0); // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + *x4 = static_cast(*s1); // expected-error {{cannot be assigned because its copy assignment operator is implicitly deleted}} + } +} + +bool test_composite_type0(bool c, int * AQ * a0, int * AQ * a1) { + auto t = c ? a0 : a1; + return a0 == a1; +} + +bool test_composite_type1(bool c, int * AQ * a0, int * AQ2 * a1) { + auto t = c ? a0 : a1; // expected-error {{incompatible operand types ('int *__ptrauth(1,1,50) *' and 'int *__ptrauth(1,1,51) *')}} + return a0 == a1; // expected-error {{comparison of distinct pointer types ('int *__ptrauth(1,1,50) *' and 'int *__ptrauth(1,1,51) *')}} +} + +void test_bad_call_diag(void *AQ *ptr); // expected-note{{candidate function not viable: 1st argument ('void *__ptrauth(1,1,51) *') has __ptrauth(1,1,51) qualifier, but parameter has __ptrauth(1,1,50) qualifier}} expected-note {{candidate function not viable: 1st argument ('void **') has no '__ptrauth' qualifier, but parameter has __ptrauth(1,1,50) qualifier}} +void test_bad_call_diag2(void **ptr); // expected-note {{candidate function not viable: 1st argument ('void *__ptrauth(1,1,50) *') has __ptrauth(1,1,50) qualifier, but parameter has no '__ptrauth' qualifier}} + +int test_call_diag() { + void *AQ ptr1, *AQ2 ptr2, *ptr3; + test_bad_call_diag(&ptr2); // expected-error {{no matching function for call to 'test_bad_call_diag'}} + test_bad_call_diag(&ptr3); // expected-error {{no matching function for call to 'test_bad_call_diag'}} + test_bad_call_diag2(&ptr1); // expected-error {{no matching function for call to 'test_bad_call_diag2'}} +} + +namespace test_constexpr { + constexpr int i = 100; + constexpr const int * AQ p = &i; + constexpr const int * const AQ *pp = &p; + constexpr int i1 = **((const int * const AQ *)pp); + constexpr int i2 = **((const int * const AQ2 *)pp); + // expected-error@-1 {{constexpr variable 'i2' must be initialized by a constant expression}} + // expected-note@-2 {{cast that performs the conversions of a reinterpret_cast is not allowed in a constant expression}} +} + +namespace test_lambda { + void test() { + int * AQ v0; + int * AQ *v1; + + [v0, v1]() { + static_assert(__is_same(decltype(v0), int * AQ)); + static_assert(__is_same(decltype(v1), int * AQ *)); + }(); + + [v2 = v0, v3 = v1]() { + static_assert(__is_same(decltype(v2), int *)); + static_assert(__is_same(decltype(v3), int * AQ *)); + }(); + } +} + +namespace test_concept { + template struct is_qualified { + static constexpr bool value = false; + }; + + template struct is_qualified { + static constexpr bool value = true; + }; + + template + concept Ptrauthable = is_qualified::value; + // expected-note@-1 2 {{because 'is_qualified::value' evaluated to false}} + // expected-note@-2 2 {{because 'is_qualified::value' evaluated to false}} + + template + requires(Ptrauthable) + struct S {}; + // expected-note@-2 {{because 'int *' does not satisfy 'Ptrauthable'}} + // expected-note@-3 {{because 'int *__ptrauth(1,1,51)' does not satisfy 'Ptrauthable'}} + + S s0; + S s1; + // expected-error@-1 {{constraints not satisfied for class template 'S' [with T = int *]}} + S s1; + // expected-error@-1 {{constraints not satisfied for class template 'S' [with T = int *__ptrauth(1,1,51)]}} + + template + requires(Ptrauthable) + void func(T *); + // expected-note@-1 {{candidate template ignored: constraints not satisfied [with T = int *]}} + // expected-note@-3 {{because 'int *' does not satisfy 'Ptrauthable'}} + // expected-note@-3 {{candidate template ignored: constraints not satisfied [with T = int *__ptrauth(1,1,51)]}} + // expected-note@-5 {{because 'int *__ptrauth(1,1,51)' does not satisfy 'Ptrauthable'}} + + void test() { + int * AQ p0; + int *p1; + int * AQ2 p2; + func(&p0); + func(&p1); // expected-error {{no matching function for call to 'func'}} + func(&p2); // expected-error {{no matching function for call to 'func'}} + } +} diff --git a/clang/test/SemaCXX/ptrauth-template-parameters.cpp b/clang/test/SemaCXX/ptrauth-template-parameters.cpp new file mode 100644 index 0000000000000..ee23d3f2ec456 --- /dev/null +++ b/clang/test/SemaCXX/ptrauth-template-parameters.cpp @@ -0,0 +1,29 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -fsyntax-only -verify -fptrauth-intrinsics -std=c++11 %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -fsyntax-only -verify -fptrauth-intrinsics -std=c++11 %s + +template struct G { + T __ptrauth(0,0,1234) test; + // expected-error@-1 2 {{type '__ptrauth(0,0,1234) T' is already __ptrauth-qualified}} +}; + +template struct Indirect { + G layers; + // expected-note@-1{{in instantiation of template class 'G' requested here}} + // expected-note@-2{{in instantiation of template class 'G' requested here}} +}; + +template +struct TemplateParameters { + void * __ptrauth(K, 0, 100) m1; // expected-error {{expression is not an integer constant expression}} + void * __ptrauth(0, A, 100) m2; // expected-error {{argument to '__ptrauth' must be an integer constant expression}} + void * __ptrauth(0, 0, D) m3; // expected-error {{argument to '__ptrauth' must be an integer constant expression}} +}; + +void f3() { + // FIXME: consider loosening the restrictions so that the first two cases are accepted. + Indirect one; + // expected-note@-1{{in instantiation of template class 'Indirect' requested here}} + Indirect two; + // expected-note@-1{{in instantiation of template class 'Indirect' requested here}} + Indirect three; +} diff --git a/clang/test/SemaObjC/ptrauth-qualifier.m b/clang/test/SemaObjC/ptrauth-qualifier.m new file mode 100644 index 0000000000000..4836a653dd02f --- /dev/null +++ b/clang/test/SemaObjC/ptrauth-qualifier.m @@ -0,0 +1,56 @@ +// RUN: %clang_cc1 -triple arm64-apple-ios -fsyntax-only -verify -fptrauth-intrinsics %s +// RUN: %clang_cc1 -triple aarch64-linux-gnu -fsyntax-only -verify -fptrauth-intrinsics %s + +#if __has_feature(ptrauth_qualifier) +#warning __ptrauth qualifier enabled! +// expected-warning@-1 {{__ptrauth qualifier enabled!}} +#endif + +@interface Foo +// expected-warning@-1 {{class 'Foo' defined without specifying a base class}} +// expected-note@-2 {{add a super class to fix this problem}} + +@property void *__ptrauth(1, 1, 1) invalid1; +// expected-error@-1 {{property may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,1,1)'}} + +@property void *__ptrauth(1, 0, 1) invalid2; +// expected-error@-1 {{property may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,0,1)'}} + +- (void *__ptrauth(1, 1, 1))invalid5; +// expected-error@-1 {{return type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,1,1)'}} + +- (void *__ptrauth(1, 0, 1))invalid6; +// expected-error@-1 {{return type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,0,1)'}} + +- (void)invalid9:(void *__ptrauth(1, 1, 1))a; +// expected-error@-1 {{parameter type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,1,1)'}} +// expected-note@-2 {{method 'invalid9:' declared here}} + +- (void)invalid10:(void *__ptrauth(1, 0, 1))a; +// expected-error@-1 {{parameter type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,0,1)'}} +// expected-note@-2 {{method 'invalid10:' declared here}} + +@end + +@implementation Foo +// expected-warning@-1 2{{method definition for}} + +- (void *__ptrauth(1, 1, 1))invalid13 { +// expected-error@-1 {{return type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,1,1)'}} + return 0; +} + +- (void *__ptrauth(1, 0, 1))invalid14 { +// expected-error@-1 {{return type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,0,1)'}} + return 0; +} + +- (void)invalid17:(void *__ptrauth(1, 1, 1))a { +// expected-error@-1 {{parameter type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,1,1)'}} +} + +- (void)invalid18:(void *__ptrauth(1, 0, 1))a { +// expected-error@-1 {{parameter type may not be qualified with '__ptrauth'; type is 'void *__ptrauth(1,0,1)'}} +} + +@end diff --git a/libcxxabi/test/test_demangle.pass.cpp b/libcxxabi/test/test_demangle.pass.cpp index abaa787f5432b..53da1bf6765e7 100644 --- a/libcxxabi/test/test_demangle.pass.cpp +++ b/libcxxabi/test/test_demangle.pass.cpp @@ -30243,6 +30243,9 @@ const char* cases[][2] = { {"_Z1fDSDRm", "f(_Sat unsigned long _Fract)"}, {"_Z11bfloat16addDF16bDF16b", "bfloat16add(std::bfloat16_t, std::bfloat16_t)"}, + + {"_Z3fooPU9__ptrauthILj3ELb1ELj234EEPi", "foo(int* __ptrauth<3u, true, 234u>*)"}, + {"_Z3fooIPU9__ptrauthILj1ELb0ELj64EEPiEvT_", "void foo*>(int* __ptrauth<1u, false, 64u>*)"}, // clang-format on }; diff --git a/llvm/include/llvm/Demangle/MicrosoftDemangle.h b/llvm/include/llvm/Demangle/MicrosoftDemangle.h index 276efa7603690..b9a25e361eec0 100644 --- a/llvm/include/llvm/Demangle/MicrosoftDemangle.h +++ b/llvm/include/llvm/Demangle/MicrosoftDemangle.h @@ -173,6 +173,14 @@ class Demangler { Qualifiers demanglePointerExtQualifiers(std::string_view &MangledName); + bool isMemberPointer(std::string_view MangledName, bool &Error); + + std::optional + demanglePointerAuthQualifier(std::string_view &MangledName); + + PointerAuthQualifierNode * + createPointerAuthQualifier(std::string_view &MangledName); + // Parser functions. This is a recursive-descent parser. TypeNode *demangleType(std::string_view &MangledName, QualifierMangleMode QMM); diff --git a/llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h b/llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h index 09b9d947464ae..d72fb47cd9b04 100644 --- a/llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h +++ b/llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h @@ -253,7 +253,8 @@ enum class NodeKind { LocalStaticGuardVariable, FunctionSymbol, VariableSymbol, - SpecialTableSymbol + SpecialTableSymbol, + PointerAuthQualifier, }; struct Node { @@ -295,6 +296,7 @@ struct SymbolNode; struct FunctionSymbolNode; struct VariableSymbolNode; struct SpecialTableSymbolNode; +struct PointerAuthQualifierNode; struct TypeNode : public Node { explicit TypeNode(NodeKind K) : Node(K) {} @@ -467,6 +469,8 @@ struct PointerTypeNode : public TypeNode { // If this is a member pointer, this is the class that the member is in. QualifiedNameNode *ClassParent = nullptr; + PointerAuthQualifierNode *PointerAuthQualifier = nullptr; + // Represents a type X in "a pointer to X", "a reference to X", or // "rvalue-reference to X" TypeNode *Pointee = nullptr; @@ -625,6 +629,22 @@ struct FunctionSymbolNode : public SymbolNode { FunctionSignatureNode *Signature = nullptr; }; +struct PointerAuthQualifierNode : public Node { + PointerAuthQualifierNode() : Node(NodeKind::PointerAuthQualifier) {} + + // __ptrauth takes three arguments: + // - key + // - isAddressDiscriminated + // - extra discriminator + static constexpr unsigned NumArgs = 3; + typedef std::array ArgArray; + + void output(OutputBuffer &OB, OutputFlags Flags) const override; + + // List of arguments. + NodeArrayNode *Components = nullptr; +}; + } // namespace ms_demangle } // namespace llvm diff --git a/llvm/lib/Demangle/MicrosoftDemangle.cpp b/llvm/lib/Demangle/MicrosoftDemangle.cpp index 8d5f6b21e2e76..b22928be3be50 100644 --- a/llvm/lib/Demangle/MicrosoftDemangle.cpp +++ b/llvm/lib/Demangle/MicrosoftDemangle.cpp @@ -66,7 +66,7 @@ static bool startsWith(std::string_view S, std::string_view PrefixA, return llvm::itanium_demangle::starts_with(S, Prefix); } -static bool isMemberPointer(std::string_view MangledName, bool &Error) { +bool Demangler::isMemberPointer(std::string_view MangledName, bool &Error) { Error = false; const char F = MangledName.front(); MangledName.remove_prefix(1); @@ -107,6 +107,7 @@ static bool isMemberPointer(std::string_view MangledName, bool &Error) { consumeFront(MangledName, 'E'); // 64-bit consumeFront(MangledName, 'I'); // restrict consumeFront(MangledName, 'F'); // unaligned + demanglePointerAuthQualifier(MangledName); if (MangledName.empty()) { Error = true; @@ -2099,6 +2100,8 @@ PointerTypeNode *Demangler::demanglePointerType(std::string_view &MangledName) { Qualifiers ExtQuals = demanglePointerExtQualifiers(MangledName); Pointer->Quals = Qualifiers(Pointer->Quals | ExtQuals); + Pointer->PointerAuthQualifier = createPointerAuthQualifier(MangledName); + Pointer->Pointee = demangleType(MangledName, QualifierMangleMode::Mangle); return Pointer; } @@ -2147,6 +2150,49 @@ Demangler::demanglePointerExtQualifiers(std::string_view &MangledName) { return Quals; } +std::optional +Demangler::demanglePointerAuthQualifier(std::string_view &MangledName) { + if (!consumeFront(MangledName, "__ptrauth")) + return std::nullopt; + + constexpr unsigned NumArgs = PointerAuthQualifierNode::NumArgs; + PointerAuthQualifierNode::ArgArray Array; + + for (unsigned I = 0; I < NumArgs; ++I) { + bool IsNegative = false; + uint64_t Value = 0; + std::tie(Value, IsNegative) = demangleNumber(MangledName); + if (IsNegative) + return std::nullopt; + + Array[I] = Value; + } + + return Array; +} + +PointerAuthQualifierNode * +Demangler::createPointerAuthQualifier(std::string_view &MangledName) { + constexpr unsigned NumArgs = PointerAuthQualifierNode::NumArgs; + std::optional Vals = + demanglePointerAuthQualifier(MangledName); + + if (!Vals) + return nullptr; + + PointerAuthQualifierNode *PtrAuthQual = + Arena.alloc(); + NodeArrayNode *Array = Arena.alloc(); + PtrAuthQual->Components = Array; + Array->Count = NumArgs; + Array->Nodes = Arena.allocArray(NumArgs); + + for (unsigned I = 0; I < NumArgs; ++I) + Array->Nodes[I] = Arena.alloc((*Vals)[I], false); + + return PtrAuthQual; +} + ArrayTypeNode *Demangler::demangleArrayType(std::string_view &MangledName) { assert(MangledName.front() == 'Y'); MangledName.remove_prefix(1); diff --git a/llvm/lib/Demangle/MicrosoftDemangleNodes.cpp b/llvm/lib/Demangle/MicrosoftDemangleNodes.cpp index ec6e67058c683..61e4961c714bc 100644 --- a/llvm/lib/Demangle/MicrosoftDemangleNodes.cpp +++ b/llvm/lib/Demangle/MicrosoftDemangleNodes.cpp @@ -521,6 +521,9 @@ void PointerTypeNode::outputPre(OutputBuffer &OB, OutputFlags Flags) const { assert(false); } outputQualifiers(OB, Quals, false, false); + + if (PointerAuthQualifier) + PointerAuthQualifier->output(OB, Flags); } void PointerTypeNode::outputPost(OutputBuffer &OB, OutputFlags Flags) const { @@ -591,6 +594,13 @@ void FunctionSymbolNode::output(OutputBuffer &OB, OutputFlags Flags) const { Signature->outputPost(OB, Flags); } +void PointerAuthQualifierNode::output(OutputBuffer &OB, + OutputFlags Flags) const { + OB << "__ptrauth("; + Components->output(OB, Flags); + OB << ")"; +} + void VariableSymbolNode::output(OutputBuffer &OB, OutputFlags Flags) const { const char *AccessSpec = nullptr; bool IsStatic = true; diff --git a/llvm/test/Demangle/ms-ptrauth.test b/llvm/test/Demangle/ms-ptrauth.test new file mode 100644 index 0000000000000..18a9f37bec67a --- /dev/null +++ b/llvm/test/Demangle/ms-ptrauth.test @@ -0,0 +1,12 @@ +; RUN: llvm-undname < %s | FileCheck %s + +; CHECK-NOT: Invalid mangled name + +?s@@3U?$S@PE__ptrauth1A@ENC@AH@@A +; CHECK: struct S s + +?foo@@YAXPEAPE__ptrauth20OK@AH@Z +; CHECK: void __cdecl foo(int *__ptrauth(3, 1, 234)*) + +??$foo@PEAPE__ptrauth0A@EA@AH@@YAXPEAPE__ptrauth0A@EA@AH@Z +; CHECK: void __cdecl foo(int *__ptrauth(1, 0, 64)*) From a1d52fcdca8bda4fe8c6652b2de83f408f4ad4f2 Mon Sep 17 00:00:00 2001 From: Peter Collingbourne Date: Tue, 15 Apr 2025 13:03:58 -0700 Subject: [PATCH 20/63] Clarify documentation for -funique-source-file-names. Reviewers: efriedma-quic, teresajohnson Reviewed By: teresajohnson, efriedma-quic Pull Request: https://github.com/llvm/llvm-project/pull/135832 --- clang/docs/UsersManual.rst | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/clang/docs/UsersManual.rst b/clang/docs/UsersManual.rst index d4656a7e63c99..69256527f40c9 100644 --- a/clang/docs/UsersManual.rst +++ b/clang/docs/UsersManual.rst @@ -2301,11 +2301,31 @@ are listed below. When enabled, allows the compiler to assume that each object file passed to the linker has been compiled using a unique source file - name. This is useful for reducing link times when doing ThinLTO + path. This is useful for reducing link times when doing ThinLTO in combination with whole-program devirtualization or CFI. - A misuse of this flag will generally result in a duplicate symbol - error at link time. + The full source path passed to the compiler must be unique. This + means that, for example, the following is a usage error: + + .. code-block:: console + + $ cd foo + $ clang -funique-source-file-names -c foo.c + $ cd ../bar + $ clang -funique-source-file-names -c foo.c + $ cd .. + $ clang foo/foo.o bar/foo.o + + but this is not: + + .. code-block:: console + + $ clang -funique-source-file-names -c foo/foo.c + $ clang -funique-source-file-names -c bar/foo.c + $ clang foo/foo.o bar/foo.o + + A misuse of this flag may result in a duplicate symbol error at + link time. .. option:: -fforce-emit-vtables From 2271f0bebd48c9ed8b16b500886a819c4f269a6a Mon Sep 17 00:00:00 2001 From: Alexey Bataev Date: Tue, 15 Apr 2025 13:02:32 -0700 Subject: [PATCH 21/63] [SLP]Check for perfect/shuffled match for the split node If the potential split node is a perfect/shuffled match of another split node, need to skip creation of the another split node with the same scalars, it should be a buildvector. Fixes #135800 --- .../Transforms/Vectorize/SLPVectorizer.cpp | 18 +++++ .../X86/split-node-full-match.ll | 74 +++++++++++++++++++ .../X86/split-node-no-reorder-copy.ll | 3 +- 3 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/split-node-full-match.ll diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp index 253933a2438cd..234cd340ebc13 100644 --- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp +++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp @@ -9575,6 +9575,24 @@ bool BoUpSLP::canBuildSplitNode(ArrayRef VL, !SplitAlternateInstructions) return false; + // Check if this is a duplicate of another split entry. + LLVM_DEBUG(dbgs() << "SLP: \tChecking bundle: " << *LocalState.getMainOp() + << ".\n"); + for (TreeEntry *E : getSplitTreeEntries(LocalState.getMainOp())) { + if (E->isSame(VL)) { + LLVM_DEBUG(dbgs() << "SLP: Perfect diamond merge at " + << *LocalState.getMainOp() << ".\n"); + return false; + } + SmallPtrSet Values(llvm::from_range, E->Scalars); + if (all_of(VL, [&](Value *V) { + return isa(V) || Values.contains(V); + })) { + LLVM_DEBUG(dbgs() << "SLP: Gathering due to full overlap.\n"); + return false; + } + } + ReorderIndices.assign(VL.size(), VL.size()); SmallBitVector Op1Indices(VL.size()); for (auto [Idx, V] : enumerate(VL)) { diff --git a/llvm/test/Transforms/SLPVectorizer/X86/split-node-full-match.ll b/llvm/test/Transforms/SLPVectorizer/X86/split-node-full-match.ll new file mode 100644 index 0000000000000..10e73b042f19b --- /dev/null +++ b/llvm/test/Transforms/SLPVectorizer/X86/split-node-full-match.ll @@ -0,0 +1,74 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5 +; RUN: opt -S --passes=slp-vectorizer -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck %s + +define void @test(double %0) { +; CHECK-LABEL: define void @test( +; CHECK-SAME: double [[TMP0:%.*]]) { +; CHECK-NEXT: [[_THREAD:.*:]] +; CHECK-NEXT: [[TMP1:%.*]] = call double null(ptr null, ptr null, ptr null) +; CHECK-NEXT: [[TMP2:%.*]] = call double null(ptr null, ptr null, ptr null) +; CHECK-NEXT: br i1 false, label %[[BB3:.*]], label %[[BB7:.*]] +; CHECK: [[BB3]]: +; CHECK-NEXT: [[TMP4:%.*]] = call double null(ptr null, ptr null, ptr null) +; CHECK-NEXT: [[TMP5:%.*]] = insertelement <4 x double> , double [[TMP0]], i32 2 +; CHECK-NEXT: [[TMP6:%.*]] = insertelement <4 x double> [[TMP5]], double [[TMP4]], i32 0 +; CHECK-NEXT: br label %[[BB7]] +; CHECK: [[BB7]]: +; CHECK-NEXT: [[TMP8:%.*]] = phi <4 x double> [ [[TMP6]], %[[BB3]] ], [ zeroinitializer, [[DOTTHREAD:%.*]] ] +; CHECK-NEXT: [[TMP9:%.*]] = insertelement <2 x double> poison, double [[TMP2]], i32 0 +; CHECK-NEXT: [[TMP10:%.*]] = insertelement <2 x double> [[TMP9]], double [[TMP1]], i32 1 +; CHECK-NEXT: [[TMP11:%.*]] = shufflevector <4 x double> [[TMP8]], <4 x double> poison, <6 x i32> +; CHECK-NEXT: [[TMP12:%.*]] = call <6 x double> @llvm.vector.insert.v6f64.v2f64(<6 x double> [[TMP11]], <2 x double> [[TMP10]], i64 4) +; CHECK-NEXT: br i1 false, label %[[DOTLR_PH272_PREHEADER:.*]], [[DOT_CRIT_EDGE:label %.*]] +; CHECK: [[_LR_PH272_PREHEADER:.*:]] +; CHECK-NEXT: br i1 false, [[DOT_CRIT_EDGE]], label %[[BB13:.*]] +; CHECK: [[BB13]]: +; CHECK-NEXT: [[TMP14:%.*]] = shufflevector <6 x double> [[TMP12]], <6 x double> poison, <4 x i32> +; CHECK-NEXT: [[TMP15:%.*]] = shufflevector <4 x double> [[TMP14]], <4 x double> poison, <6 x i32> +; CHECK-NEXT: [[TMP16:%.*]] = call <6 x double> @llvm.vector.insert.v6f64.v2f64(<6 x double> [[TMP15]], <2 x double> splat (double 0x7FF8000000000000), i64 4) +; CHECK-NEXT: br i1 false, label %[[BB17:.*]], [[DOT_CRIT_EDGE]] +; CHECK: [[BB17]]: +; CHECK-NEXT: [[TMP18:%.*]] = insertelement <6 x double> , double [[TMP0]], i32 3 +; CHECK-NEXT: br [[DOT_CRIT_EDGE]] +; CHECK: [[__CRIT_EDGE:.*:]] +; CHECK-NEXT: [[TMP19:%.*]] = phi <6 x double> [ [[TMP12]], %[[BB7]] ], [ [[TMP18]], %[[BB17]] ], [ [[TMP16]], %[[BB13]] ], [ [[TMP12]], %[[DOTLR_PH272_PREHEADER]] ] +; CHECK-NEXT: ret void +; +.thread: + %1 = call double null(ptr null, ptr null, ptr null) + %2 = call double null(ptr null, ptr null, ptr null) + br i1 false, label %3, label %5 + +3: + %4 = call double null(ptr null, ptr null, ptr null) + br label %5 + +5: + %.1226 = phi double [ %4, %3 ], [ 0.000000e+00, %.thread ] + %.1222 = phi double [ 0.000000e+00, %3 ], [ 0.000000e+00, %.thread ] + %.1218 = phi double [ %0, %3 ], [ 0.000000e+00, %.thread ] + %.1216 = phi double [ 0.000000e+00, %3 ], [ 0.000000e+00, %.thread ] + br i1 false, label %.lr.ph272.preheader, label %._crit_edge + +.lr.ph272.preheader: + br i1 false, label %._crit_edge, label %6 + +6: + %7 = fdiv double 0.000000e+00, 0.000000e+00 + %8 = fsub double 0.000000e+00, %7 + %9 = fdiv double 0.000000e+00, 0.000000e+00 + %10 = fsub double 0.000000e+00, %9 + br i1 false, label %11, label %._crit_edge + +11: + br label %._crit_edge + +._crit_edge: + %.2227.lcssa = phi double [ %.1226, %5 ], [ 0.000000e+00, %11 ], [ %.1226, %6 ], [ %.1226, %.lr.ph272.preheader ] + %.2223.lcssa = phi double [ %.1222, %5 ], [ 0.000000e+00, %11 ], [ %.1222, %6 ], [ %.1222, %.lr.ph272.preheader ] + %.2219.lcssa = phi double [ %.1218, %5 ], [ 0.000000e+00, %11 ], [ %.1218, %6 ], [ %.1218, %.lr.ph272.preheader ] + %.2.lcssa = phi double [ %.1216, %5 ], [ %0, %11 ], [ %.1216, %6 ], [ %.1216, %.lr.ph272.preheader ] + %.0213.lcssa = phi double [ %2, %5 ], [ 0.000000e+00, %11 ], [ %10, %6 ], [ %2, %.lr.ph272.preheader ] + %.0211.lcssa = phi double [ %1, %5 ], [ 0.000000e+00, %11 ], [ %8, %6 ], [ %1, %.lr.ph272.preheader ] + ret void +} diff --git a/llvm/test/Transforms/SLPVectorizer/X86/split-node-no-reorder-copy.ll b/llvm/test/Transforms/SLPVectorizer/X86/split-node-no-reorder-copy.ll index b7b6c10137b64..9abb994db1e73 100644 --- a/llvm/test/Transforms/SLPVectorizer/X86/split-node-no-reorder-copy.ll +++ b/llvm/test/Transforms/SLPVectorizer/X86/split-node-no-reorder-copy.ll @@ -15,8 +15,9 @@ define i1 @test(ptr %0, ptr %1, <2 x float> %2, <2 x float> %3, <2 x float> %4) ; CHECK-NEXT: [[TMP14:%.*]] = shufflevector <2 x float> [[TMP2]], <2 x float> [[TMP3]], <8 x i32> ; CHECK-NEXT: [[TMP15:%.*]] = insertelement <8 x float> [[TMP14]], float [[TMP9]], i32 7 ; CHECK-NEXT: [[TMP16:%.*]] = shufflevector <8 x float> [[TMP13]], <8 x float> poison, <16 x i32> -; CHECK-NEXT: [[TMP17:%.*]] = call <16 x float> @llvm.vector.insert.v16f32.v8f32(<16 x float> [[TMP16]], <8 x float> [[TMP15]], i64 8) ; CHECK-NEXT: [[TMP18:%.*]] = call <16 x float> @llvm.vector.insert.v16f32.v8f32(<16 x float> [[TMP16]], <8 x float> [[TMP15]], i64 8) +; CHECK-NEXT: [[TMP19:%.*]] = shufflevector <8 x float> [[TMP14]], <8 x float> [[TMP12]], <16 x i32> +; CHECK-NEXT: [[TMP17:%.*]] = insertelement <16 x float> [[TMP19]], float [[TMP9]], i32 15 ; CHECK-NEXT: [[TMP20:%.*]] = fmul <16 x float> [[TMP18]], [[TMP17]] ; CHECK-NEXT: [[TMP21:%.*]] = call reassoc nsz float @llvm.vector.reduce.fadd.v16f32(float 0.000000e+00, <16 x float> [[TMP20]]) ; CHECK-NEXT: [[TMP22:%.*]] = call float @foo(float [[TMP21]]) From 823adc7a2dc90cdd0f953f3dc9684481368f2b62 Mon Sep 17 00:00:00 2001 From: YongKang Zhu Date: Tue, 15 Apr 2025 13:19:15 -0700 Subject: [PATCH 22/63] [BOLT] Validate secondary entry point (#135731) Some functions have their sizes as zero in input binary's symbol table, like those compiled by assembler. When figuring out function sizes, we may create label symbol if it doesn't point to any constant island. However, before function size is known, marker symbol can not be correctly associated to a function and therefore all such checks would fail and we could end up adding a code label pointing to constant island as secondary entry point and later mistakenly marking the function as not simple. Querying the global marker symbol array has big throughput overhead. Instead we can run an extra check when post processing entry points to identify such label symbols that actually point to constant islands. --- bolt/include/bolt/Core/BinaryFunction.h | 9 +++++ bolt/lib/Core/BinaryFunction.cpp | 9 +++++ .../AArch64/validate-secondary-entry-point.s | 34 +++++++++++++++++++ .../RISCV/validate-secondary-entry-point.s | 34 +++++++++++++++++++ 4 files changed, 86 insertions(+) create mode 100644 bolt/test/AArch64/validate-secondary-entry-point.s create mode 100644 bolt/test/RISCV/validate-secondary-entry-point.s diff --git a/bolt/include/bolt/Core/BinaryFunction.h b/bolt/include/bolt/Core/BinaryFunction.h index d3d11f8c5fb73..a52998564ee1b 100644 --- a/bolt/include/bolt/Core/BinaryFunction.h +++ b/bolt/include/bolt/Core/BinaryFunction.h @@ -1174,6 +1174,11 @@ class BinaryFunction { return getSecondaryEntryPointSymbol(BB.getLabel()); } + /// Remove a label from the secondary entry point map. + void removeSymbolFromSecondaryEntryPointMap(const MCSymbol *Label) { + SecondaryEntryPoints.erase(Label); + } + /// Return true if the basic block is an entry point into the function /// (either primary or secondary). bool isEntryPoint(const BinaryBasicBlock &BB) const { @@ -2126,6 +2131,10 @@ class BinaryFunction { return Islands && !Islands->DataOffsets.empty(); } + bool isStartOfConstantIsland(uint64_t Offset) const { + return hasConstantIsland() && Islands->DataOffsets.count(Offset); + } + /// Return true iff the symbol could be seen inside this function otherwise /// it is probably another function. bool isSymbolValidInScope(const SymbolRef &Symbol, uint64_t SymbolSize) const; diff --git a/bolt/lib/Core/BinaryFunction.cpp b/bolt/lib/Core/BinaryFunction.cpp index c4f4d234b30c0..184a4462b356a 100644 --- a/bolt/lib/Core/BinaryFunction.cpp +++ b/bolt/lib/Core/BinaryFunction.cpp @@ -1896,6 +1896,15 @@ void BinaryFunction::postProcessEntryPoints() { if (BC.isAArch64() && Offset == getSize()) continue; + // If we have grabbed a wrong code label which actually points to some + // constant island inside the function, ignore this label and remove it + // from the secondary entry point map. + if (isStartOfConstantIsland(Offset)) { + BC.SymbolToFunctionMap.erase(Label); + removeSymbolFromSecondaryEntryPointMap(Label); + continue; + } + BC.errs() << "BOLT-WARNING: reference in the middle of instruction " "detected in function " << *this << " at offset 0x" << Twine::utohexstr(Offset) << '\n'; diff --git a/bolt/test/AArch64/validate-secondary-entry-point.s b/bolt/test/AArch64/validate-secondary-entry-point.s new file mode 100644 index 0000000000000..0099a0ee4fe99 --- /dev/null +++ b/bolt/test/AArch64/validate-secondary-entry-point.s @@ -0,0 +1,34 @@ +# This test is to verify that BOLT won't take a label pointing to constant +# island as a secondary entry point (function `_start` doesn't have ELF size +# set originally) and the function won't otherwise be mistaken as non-simple. + +# RUN: %clang %cflags -pie %s -o %t.so -Wl,-q -Wl,--init=_foo -Wl,--fini=_foo +# RUN: llvm-bolt %t.so -o %t.bolt.so --print-cfg 2>&1 | FileCheck %s +# CHECK-NOT: BOLT-WARNING: reference in the middle of instruction detected \ +# CHECK-NOT: function _start at offset 0x{{[0-9a-f]+}} +# CHECK: Binary Function "_start" after building cfg + + .text + + .global _foo + .type _foo, %function +_foo: + ret + + .global _start + .type _start, %function +_start: + b _foo + + .balign 16 +_random_consts: + .long 0x12345678 + .long 0x90abcdef + + .global _bar + .type _bar, %function +_bar: + ret + + # Dummy relocation to force relocation mode + .reloc 0, R_AARCH64_NONE diff --git a/bolt/test/RISCV/validate-secondary-entry-point.s b/bolt/test/RISCV/validate-secondary-entry-point.s new file mode 100644 index 0000000000000..0c29f5c97c689 --- /dev/null +++ b/bolt/test/RISCV/validate-secondary-entry-point.s @@ -0,0 +1,34 @@ +# This test is to verify that BOLT won't take a label pointing to constant +# island as a secondary entry point (function `_start` doesn't have ELF size +# set originally) and the function won't otherwise be mistaken as non-simple. + +# RUN: %clang %cflags -pie %s -o %t.so -Wl,-q -Wl,--init=_foo -Wl,--fini=_foo +# RUN: llvm-bolt %t.so -o %t.bolt.so --print-cfg 2>&1 | FileCheck %s +# CHECK-NOT: BOLT-WARNING: reference in the middle of instruction detected \ +# CHECK-NOT: function _start at offset 0x{{[0-9a-f]+}} +# CHECK: Binary Function "_start" after building cfg + + .text + + .global _foo + .type _foo, %function +_foo: + ret + + .global _start + .type _start, %function +_start: + j _foo + + .balign 16 +_random_consts: + .long 0x12345678 + .long 0x90abcdef + + .global _bar + .type _bar, %function +_bar: + ret + + # Dummy relocation to force relocation mode + .reloc 0, R_RISCV_NONE From 14cb6566d6701feaef2ffd686af5de4ff9e3eb29 Mon Sep 17 00:00:00 2001 From: John Harrison Date: Tue, 15 Apr 2025 13:30:33 -0700 Subject: [PATCH 23/63] [lldb-dap] Improve error reporting for dap command arguments. (#135684) Previously the error only contained the failed to parse JSON message, which has no additional context. This improves the error messages and improves the consistency of handling properties in protocol structures. Updating the fields to use 'ObjectMapper.map' instead of 'ObjectMapper.mapOptional' caught that adapterID was misspelled as well. For example, previously: ``` $ echo 'Content-Length: 81\r\n\r\n{"type":"request","command":"initialize","seq":1,"arguments":{"adapterID":12345}} | lldb-dap ``` Worked without an error but now it reports: ``` invalid arguments for request 'initialize': expected string at arguments.adapterID { "adapterID": /* error: expected string */ 12345 } ``` --- lldb/tools/lldb-dap/Handler/RequestHandler.h | 8 ++-- lldb/tools/lldb-dap/Protocol/ProtocolBase.cpp | 2 +- .../lldb-dap/Protocol/ProtocolRequests.cpp | 42 ++++++++++++------- .../lldb-dap/Protocol/ProtocolRequests.h | 4 +- .../tools/lldb-dap/Protocol/ProtocolTypes.cpp | 6 +-- 5 files changed, 37 insertions(+), 25 deletions(-) diff --git a/lldb/tools/lldb-dap/Handler/RequestHandler.h b/lldb/tools/lldb-dap/Handler/RequestHandler.h index 50795f8252de3..7e56c258ad78a 100644 --- a/lldb/tools/lldb-dap/Handler/RequestHandler.h +++ b/lldb/tools/lldb-dap/Handler/RequestHandler.h @@ -120,11 +120,13 @@ class RequestHandler : public BaseRequestHandler { } Args arguments; - llvm::json::Path::Root root; - if (request.arguments && !fromJSON(request.arguments, arguments, root)) { + llvm::json::Path::Root root("arguments"); + if (request.arguments && !fromJSON(*request.arguments, arguments, root)) { std::string parse_failure; llvm::raw_string_ostream OS(parse_failure); - root.printErrorContext(request.arguments, OS); + OS << "invalid arguments for request '" << request.command + << "': " << llvm::toString(root.getError()) << "\n"; + root.printErrorContext(*request.arguments, OS); protocol::ErrorMessage error_message; error_message.format = parse_failure; diff --git a/lldb/tools/lldb-dap/Protocol/ProtocolBase.cpp b/lldb/tools/lldb-dap/Protocol/ProtocolBase.cpp index af63cc803e545..bfd68448fb483 100644 --- a/lldb/tools/lldb-dap/Protocol/ProtocolBase.cpp +++ b/lldb/tools/lldb-dap/Protocol/ProtocolBase.cpp @@ -178,7 +178,7 @@ bool fromJSON(json::Value const &Params, Response &R, json::Path P) { return false; } - return O.map("success", R.success) && O.mapOptional("message", R.message) && + return O.map("success", R.success) && O.map("message", R.message) && mapRaw(Params, "body", R.body, P); } diff --git a/lldb/tools/lldb-dap/Protocol/ProtocolRequests.cpp b/lldb/tools/lldb-dap/Protocol/ProtocolRequests.cpp index 7163399899f7e..3523f8ac87ec9 100644 --- a/lldb/tools/lldb-dap/Protocol/ProtocolRequests.cpp +++ b/lldb/tools/lldb-dap/Protocol/ProtocolRequests.cpp @@ -20,16 +20,16 @@ namespace lldb_dap::protocol { bool fromJSON(const llvm::json::Value &Params, CancelArguments &CA, llvm::json::Path P) { llvm::json::ObjectMapper O(Params, P); - return O && O.mapOptional("requestId", CA.requestId) && - O.mapOptional("progressId", CA.progressId); + return O && O.map("requestId", CA.requestId) && + O.map("progressId", CA.progressId); } bool fromJSON(const json::Value &Params, DisconnectArguments &DA, json::Path P) { json::ObjectMapper O(Params, P); - return O && O.mapOptional("restart", DA.restart) && - O.mapOptional("terminateDebuggee", DA.terminateDebuggee) && - O.mapOptional("suspendDebuggee", DA.suspendDebuggee); + return O && O.map("restart", DA.restart) && + O.map("terminateDebuggee", DA.terminateDebuggee) && + O.map("suspendDebuggee", DA.suspendDebuggee); } bool fromJSON(const llvm::json::Value &Params, PathFormat &PF, @@ -75,23 +75,33 @@ bool fromJSON(const llvm::json::Value &Params, InitializeRequestArguments &IRA, const json::Object *O = Params.getAsObject(); - for (auto &kv : ClientFeatureByKey) - if (std::optional v = O->getBoolean(kv.first()); v && *v) + for (auto &kv : ClientFeatureByKey) { + const json::Value *value_ref = O->get(kv.first()); + if (!value_ref) + continue; + + const std::optional value = value_ref->getAsBoolean(); + if (!value) { + P.field(kv.first()).report("expected bool"); + return false; + } + + if (*value) IRA.supportedFeatures.insert(kv.second); + } - return OM.mapOptional("adatperID", IRA.adatperID) && - OM.mapOptional("clientID", IRA.clientID) && - OM.mapOptional("clientName", IRA.clientName) && - OM.mapOptional("locale", IRA.locale) && - OM.mapOptional("linesStartAt1", IRA.linesStartAt1) && - OM.mapOptional("columnsStartAt1", IRA.columnsStartAt1) && - OM.mapOptional("pathFormat", IRA.pathFormat) && - OM.mapOptional("$__lldb_sourceInitFile", IRA.lldbExtSourceInitFile); + return OM.map("adapterID", IRA.adapterID) && + OM.map("clientID", IRA.clientID) && + OM.map("clientName", IRA.clientName) && OM.map("locale", IRA.locale) && + OM.map("linesStartAt1", IRA.linesStartAt1) && + OM.map("columnsStartAt1", IRA.columnsStartAt1) && + OM.map("pathFormat", IRA.pathFormat) && + OM.map("$__lldb_sourceInitFile", IRA.lldbExtSourceInitFile); } bool fromJSON(const json::Value &Params, SourceArguments &SA, json::Path P) { json::ObjectMapper O(Params, P); - return O && O.mapOptional("source", SA.source) && + return O && O.map("source", SA.source) && O.map("sourceReference", SA.sourceReference); } diff --git a/lldb/tools/lldb-dap/Protocol/ProtocolRequests.h b/lldb/tools/lldb-dap/Protocol/ProtocolRequests.h index 22d400fd494a5..6623dfa0db05c 100644 --- a/lldb/tools/lldb-dap/Protocol/ProtocolRequests.h +++ b/lldb/tools/lldb-dap/Protocol/ProtocolRequests.h @@ -100,7 +100,7 @@ enum PathFormat : unsigned { ePatFormatPath, ePathFormatURI }; /// Arguments for `initialize` request. struct InitializeRequestArguments { /// The ID of the debug adapter. - std::string adatperID; + std::string adapterID; /// The ID of the client using this adapter. std::optional clientID; @@ -113,7 +113,7 @@ struct InitializeRequestArguments { /// Determines in what format paths are specified. The default is `path`, /// which is the native format. - std::optional pathFormat = ePatFormatPath; + PathFormat pathFormat = ePatFormatPath; /// If true all line numbers are 1-based (default). std::optional linesStartAt1; diff --git a/lldb/tools/lldb-dap/Protocol/ProtocolTypes.cpp b/lldb/tools/lldb-dap/Protocol/ProtocolTypes.cpp index f4f0bf8dcea84..4d1e90215bbb4 100644 --- a/lldb/tools/lldb-dap/Protocol/ProtocolTypes.cpp +++ b/lldb/tools/lldb-dap/Protocol/ProtocolTypes.cpp @@ -38,9 +38,9 @@ bool fromJSON(const json::Value &Params, PresentationHint &PH, json::Path P) { bool fromJSON(const json::Value &Params, Source &S, json::Path P) { json::ObjectMapper O(Params, P); - return O && O.mapOptional("name", S.name) && O.mapOptional("path", S.path) && - O.mapOptional("presentationHint", S.presentationHint) && - O.mapOptional("sourceReference", S.sourceReference); + return O && O.map("name", S.name) && O.map("path", S.path) && + O.map("presentationHint", S.presentationHint) && + O.map("sourceReference", S.sourceReference); } json::Value toJSON(const ExceptionBreakpointsFilter &EBF) { From 85eb44e304e0a0a7da78448ceee60fdfec235edb Mon Sep 17 00:00:00 2001 From: Alexey Bataev Date: Tue, 15 Apr 2025 13:29:11 -0700 Subject: [PATCH 24/63] [SLP]Fix number of operands for the split node FOr the split node number of operands should be requested via getNumOperands() function, even if the main op is CallInst. --- .../Transforms/Vectorize/SLPVectorizer.cpp | 2 + .../X86/split-node-num-operands.ll | 121 ++++++++++++++++++ 2 files changed, 123 insertions(+) create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/split-node-num-operands.ll diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp index 234cd340ebc13..b174f0f03fca6 100644 --- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp +++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp @@ -7577,6 +7577,8 @@ void BoUpSLP::reorderBottomToTop(bool IgnoreReorder) { return Res.takeVector(); }; auto GetNumOperands = [](const TreeEntry *TE) { + if (TE->State == TreeEntry::SplitVectorize) + return TE->getNumOperands(); if (auto *CI = dyn_cast(TE->getMainOp()); CI) return CI->arg_size(); return TE->getNumOperands(); diff --git a/llvm/test/Transforms/SLPVectorizer/X86/split-node-num-operands.ll b/llvm/test/Transforms/SLPVectorizer/X86/split-node-num-operands.ll new file mode 100644 index 0000000000000..5aa4dba2b8a1b --- /dev/null +++ b/llvm/test/Transforms/SLPVectorizer/X86/split-node-num-operands.ll @@ -0,0 +1,121 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5 +; RUN: opt -S --passes=slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -mattr=+avx -slp-threshold=-1000 < %s | FileCheck %s + +define i64 @Foo(ptr align 8 dereferenceable(344) %0, i64 %1) { +; CHECK-LABEL: define i64 @Foo( +; CHECK-SAME: ptr align 8 dereferenceable(344) [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0:[0-9]+]] { +; CHECK-NEXT: [[TMP3:%.*]] = getelementptr i8, ptr [[TMP0]], i64 104 +; CHECK-NEXT: [[TMP4:%.*]] = getelementptr i8, ptr [[TMP0]], i64 112 +; CHECK-NEXT: [[TMP5:%.*]] = getelementptr i8, ptr [[TMP0]], i64 24 +; CHECK-NEXT: [[TMP6:%.*]] = load i64, ptr [[TMP3]], align 8 +; CHECK-NEXT: [[TMP7:%.*]] = load i64, ptr [[TMP4]], align 8 +; CHECK-NEXT: [[TMP8:%.*]] = load i64, ptr [[TMP5]], align 8 +; CHECK-NEXT: [[TMP9:%.*]] = load i64, ptr [[TMP0]], align 8 +; CHECK-NEXT: [[TMP10:%.*]] = insertelement <2 x i64> poison, i64 [[TMP6]], i32 0 +; CHECK-NEXT: [[TMP11:%.*]] = insertelement <2 x i64> [[TMP10]], i64 [[TMP9]], i32 1 +; CHECK-NEXT: [[TMP12:%.*]] = insertelement <2 x i64> poison, i64 [[TMP7]], i32 0 +; CHECK-NEXT: [[TMP13:%.*]] = insertelement <2 x i64> [[TMP12]], i64 [[TMP8]], i32 1 +; CHECK-NEXT: [[TMP14:%.*]] = insertelement <2 x i64> poison, i64 0, i32 0 +; CHECK-NEXT: [[TMP15:%.*]] = insertelement <2 x i64> , i64 [[TMP1]], i32 1 +; CHECK-NEXT: br label %[[BB16:.*]] +; CHECK: [[BB16]]: +; CHECK-NEXT: [[TMP17:%.*]] = phi <2 x i64> [ [[TMP11]], [[TMP2:%.*]] ], [ zeroinitializer, %[[TMP25:.*]] ] +; CHECK-NEXT: [[TMP18:%.*]] = phi <2 x i64> [ [[TMP13]], [[TMP2]] ], [ [[TMP29:%.*]], %[[TMP25]] ] +; CHECK-NEXT: switch i32 0, label %[[BB19:.*]] [ +; CHECK-NEXT: i32 0, label %[[TMP25]] +; CHECK-NEXT: ] +; CHECK: [[BB19]]: +; CHECK-NEXT: [[TMP20:%.*]] = shufflevector <2 x i64> [[TMP18]], <2 x i64> poison, <4 x i32> +; CHECK-NEXT: [[TMP21:%.*]] = insertelement <4 x i64> [[TMP20]], i64 0, i32 1 +; CHECK-NEXT: [[TMP22:%.*]] = insertelement <4 x i64> [[TMP21]], i64 0, i32 2 +; CHECK-NEXT: [[TMP23:%.*]] = shufflevector <4 x i64> [[TMP22]], <4 x i64> poison, <4 x i32> +; CHECK-NEXT: [[TMP24:%.*]] = shufflevector <2 x i64> [[TMP14]], <2 x i64> [[TMP18]], <2 x i32> +; CHECK-NEXT: br label %[[TMP25]] +; CHECK: [[TMP25]]: +; CHECK-NEXT: [[TMP26:%.*]] = phi <2 x i64> [ [[TMP17]], %[[BB19]] ], [ zeroinitializer, %[[BB16]] ] +; CHECK-NEXT: [[TMP27:%.*]] = phi <4 x i64> [ [[TMP23]], %[[BB19]] ], [ zeroinitializer, %[[BB16]] ] +; CHECK-NEXT: [[TMP28:%.*]] = phi <2 x i64> [ [[TMP24]], %[[BB19]] ], [ [[TMP15]], %[[BB16]] ] +; CHECK-NEXT: [[TMP29]] = shufflevector <2 x i64> [[TMP18]], <2 x i64> , <2 x i32> +; CHECK-NEXT: br i1 false, label %[[DOTLOOPEXIT206:.*]], label %[[BB16]] +; CHECK: [[_LOOPEXIT206:.*:]] +; CHECK-NEXT: switch i32 0, label %[[BB32:.*]] [ +; CHECK-NEXT: i32 0, [[DOTCONT174:label %.*]] +; CHECK-NEXT: i32 1, label %[[BB30:.*]] +; CHECK-NEXT: ] +; CHECK: [[BB30]]: +; CHECK-NEXT: [[TMP31:%.*]] = shufflevector <4 x i64> [[TMP27]], <4 x i64> , <4 x i32> +; CHECK-NEXT: br [[DOTCONT174]] +; CHECK: [[BB32]]: +; CHECK-NEXT: [[TMP33:%.*]] = insertelement <4 x i64> [[TMP27]], i64 0, i32 1 +; CHECK-NEXT: [[TMP34:%.*]] = insertelement <4 x i64> [[TMP33]], i64 0, i32 2 +; CHECK-NEXT: [[TMP35:%.*]] = shufflevector <4 x i64> [[TMP34]], <4 x i64> poison, <4 x i32> +; CHECK-NEXT: [[TMP36:%.*]] = insertelement <2 x i64> [[TMP28]], i64 0, i32 0 +; CHECK-NEXT: br [[DOTCONT174]] +; CHECK: [[_CONT174:.*:]] +; CHECK-NEXT: [[TMP37:%.*]] = phi <2 x i64> [ [[TMP26]], %[[BB32]] ], [ zeroinitializer, %[[BB30]] ], [ [[TMP26]], %[[DOTLOOPEXIT206]] ] +; CHECK-NEXT: [[TMP38:%.*]] = phi <4 x i64> [ [[TMP35]], %[[BB32]] ], [ [[TMP31]], %[[BB30]] ], [ [[TMP27]], %[[DOTLOOPEXIT206]] ] +; CHECK-NEXT: [[TMP39:%.*]] = phi <2 x i64> [ [[TMP36]], %[[BB32]] ], [ zeroinitializer, %[[BB30]] ], [ [[TMP28]], %[[DOTLOOPEXIT206]] ] +; CHECK-NEXT: ret i64 0 +; + %3 = getelementptr i8, ptr %0, i64 104 + %4 = getelementptr i8, ptr %0, i64 112 + %5 = getelementptr i8, ptr %0, i64 24 + %6 = load i64, ptr %3, align 8 + %7 = load i64, ptr %4, align 8 + %8 = load i64, ptr %5, align 8 + %9 = load i64, ptr %0, align 8 + br label %10 + +10: + %11 = phi i64 [ %9, %2 ], [ 0, %18 ] + %12 = phi i64 [ %8, %2 ], [ %12, %18 ] + %13 = phi i64 [ %7, %2 ], [ 0, %18 ] + %14 = phi i64 [ %6, %2 ], [ 0, %18 ] + switch i32 0, label %15 [ + i32 0, label %18 + ] + +15: + %16 = tail call i64 @llvm.umin.i64(i64 0, i64 0) + %17 = tail call i64 @llvm.umax.i64(i64 0, i64 0) + br label %18 + +18: + %19 = phi i64 [ %17, %15 ], [ 0, %10 ] + %20 = phi i64 [ %16, %15 ], [ 0, %10 ] + %21 = phi i64 [ %11, %15 ], [ 0, %10 ] + %22 = phi i64 [ %12, %15 ], [ 0, %10 ] + %23 = phi i64 [ %13, %15 ], [ %1, %10 ] + %24 = phi i64 [ %14, %15 ], [ 0, %10 ] + br i1 false, label %.loopexit206, label %10 + +.loopexit206: + switch i32 0, label %26 [ + i32 0, label %.cont174 + i32 1, label %25 + ] + +25: + br label %.cont174 + +26: + %27 = tail call i64 @llvm.umin.i64(i64 0, i64 0) + %28 = tail call i64 @llvm.umax.i64(i64 0, i64 0) + br label %.cont174 + +.cont174: + %.sroa.139.1 = phi i64 [ %28, %26 ], [ %19, %25 ], [ %19, %.loopexit206 ] + %.sroa.133.1 = phi i64 [ %27, %26 ], [ 0, %25 ], [ %20, %.loopexit206 ] + %.sroa.81.1 = phi i64 [ %23, %26 ], [ 0, %25 ], [ %23, %.loopexit206 ] + %.sroa.75.1 = phi i64 [ %24, %26 ], [ 0, %25 ], [ %24, %.loopexit206 ] + %.sroa.21.1 = phi i64 [ %21, %26 ], [ 0, %25 ], [ %21, %.loopexit206 ] + %.sroa.15.1 = phi i64 [ %22, %26 ], [ 0, %25 ], [ %22, %.loopexit206 ] + %29 = phi i64 [ %28, %26 ], [ 0, %25 ], [ %19, %.loopexit206 ] + %30 = phi i64 [ %27, %26 ], [ 0, %25 ], [ %20, %.loopexit206 ] + ret i64 0 +} + +declare i64 @llvm.umax.i64(i64, i64) + +declare i64 @llvm.umin.i64(i64, i64) + From 2b983a24583dd4e131d727717872a56712b5dd52 Mon Sep 17 00:00:00 2001 From: Zhuoran Yin Date: Tue, 15 Apr 2025 16:36:25 -0400 Subject: [PATCH 25/63] [MLIR][AMDGPU] Adding dynamic size check to avoid subword buffer load (#135014) Motivation: amdgpu buffer load instruction will return all zeros when loading sub-word values. For example, assuming the buffer size is exactly one word and we attempt to invoke `llvm.amdgcn.raw.ptr.buffer.load.v2i32` starting from byte 2 of the word, we will not receive the actual value of the buffer but all zeros for the first word. This is because the boundary has been crossed for the first word. This PR come up with a fix to this problem, such that, it creates a bounds check against the buffer load instruction. It will compare the offset + vector size to see if the upper bound of the address will exceed the buffer size. If it does, masked transfer read will be optimized to `vector.load` + `arith.select`, else, it will continue to fall back to default lowering of the masked vector load. --- .../mlir/Dialect/AMDGPU/Transforms/Passes.td | 13 +- .../Dialect/AMDGPU/Transforms/CMakeLists.txt | 1 + .../AMDGPU/Transforms/TransferReadToLoad.cpp | 159 ++++++++++++++++-- .../Dialect/AMDGPU/transfer-read-to-load.mlir | 94 +++++++++-- .../llvm-project-overlay/mlir/BUILD.bazel | 1 + 5 files changed, 233 insertions(+), 35 deletions(-) diff --git a/mlir/include/mlir/Dialect/AMDGPU/Transforms/Passes.td b/mlir/include/mlir/Dialect/AMDGPU/Transforms/Passes.td index 761caa448a57c..0e858108acf35 100644 --- a/mlir/include/mlir/Dialect/AMDGPU/Transforms/Passes.td +++ b/mlir/include/mlir/Dialect/AMDGPU/Transforms/Passes.td @@ -54,15 +54,20 @@ def AmdgpuResolveStridedMetadataPass : Pass<"amdgpu-resolve-strided-metadata"> { def AmdgpuTransferReadToLoadPass : Pass<"amdgpu-transfer-read-to-load"> { let summary = "Lower the operations from the vector transfer_read to vector load"; let description = [{ - This pass creates a transfer read op lowering. A vector trasfer read op - will be lowered to a combination of vector.load, arith.select and - vector.broadcast. + This pass creates a transfer read op lowering optimization. The lowering + will produce a conditional check at runtime. If within bounds, a vector + trasfer read op will be lowered to a combination of vector.load, arith.select + and vector.broadcast. If not, it will fallback to the default lowering + of the transfer_read op. This pattern will make it possible for masked transfer_read to be lowered towards buffer load with bounds check, allowing a more optimized global load accessing pattern compared with existing implementation of llvm.intr.masked.load on vectors. }]; - let dependentDialects = []; + let dependentDialects = [ + "scf::SCFDialect", + "memref::MemRefDialect" + ]; } #endif // MLIR_DIALECT_AMDGPU_TRANSFORMS_PASSES_TD_ diff --git a/mlir/lib/Dialect/AMDGPU/Transforms/CMakeLists.txt b/mlir/lib/Dialect/AMDGPU/Transforms/CMakeLists.txt index bc5b6e9186449..8709a27e0168e 100644 --- a/mlir/lib/Dialect/AMDGPU/Transforms/CMakeLists.txt +++ b/mlir/lib/Dialect/AMDGPU/Transforms/CMakeLists.txt @@ -14,6 +14,7 @@ add_mlir_dialect_library(MLIRAMDGPUTransforms MLIRAMDGPUUtils MLIRArithDialect MLIRMemRefDialect + MLIRSCFDialect MLIRVectorDialect MLIRControlFlowDialect MLIRFuncDialect diff --git a/mlir/lib/Dialect/AMDGPU/Transforms/TransferReadToLoad.cpp b/mlir/lib/Dialect/AMDGPU/Transforms/TransferReadToLoad.cpp index 3c1a2eb962037..f665c1794cdd4 100644 --- a/mlir/lib/Dialect/AMDGPU/Transforms/TransferReadToLoad.cpp +++ b/mlir/lib/Dialect/AMDGPU/Transforms/TransferReadToLoad.cpp @@ -9,13 +9,22 @@ #include "mlir/Dialect/AMDGPU/Transforms/Passes.h" #include "mlir/Dialect/AMDGPU/IR/AMDGPUDialect.h" +#include "mlir/Dialect/Affine/IR/AffineOps.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/Arith/Utils/Utils.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/MemRef/Utils/MemRefUtils.h" +#include "mlir/Dialect/SCF/IR/SCF.h" #include "mlir/Dialect/Vector/IR/VectorOps.h" +#include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h" #include "mlir/IR/BuiltinTypes.h" +#include "mlir/IR/OpDefinition.h" #include "mlir/IR/PatternMatch.h" #include "mlir/IR/TypeUtilities.h" #include "mlir/Pass/Pass.h" #include "mlir/Support/LogicalResult.h" -#include "mlir/Transforms/WalkPatternRewriteDriver.h" +#include "mlir/Transforms/GreedyPatternRewriteDriver.h" +#include "llvm/Support/MathExtras.h" namespace mlir::amdgpu { #define GEN_PASS_DEF_AMDGPUTRANSFERREADTOLOADPASS @@ -67,6 +76,9 @@ static LogicalResult transferPreconditions( if (!memRefType.isLastDimUnitStride()) return rewriter.notifyMatchFailure(xferOp, "!= 1 stride needs VectorToSCF"); + if (memRefType.getElementTypeBitWidth() < 8) + return rewriter.notifyMatchFailure(xferOp, "unsupported sub-byte type"); + // If there is broadcasting involved then we first load the unbroadcasted // vector, and then broadcast it with `vector.broadcast`. ArrayRef vectorShape = xferOp.getVectorType().getShape(); @@ -101,6 +113,26 @@ static LogicalResult transferPreconditions( return success(); } +static Value createVectorLoadForMaskedLoad(OpBuilder &builder, Location loc, + vector::TransferReadOp readOp, + bool requiresBroadcasting, + VectorType unbroadcastedVectorType) { + Value fill = builder.create(loc, unbroadcastedVectorType, + readOp.getPadding()); + Value load = builder.create( + loc, unbroadcastedVectorType, readOp.getSource(), readOp.getIndices()); + Value res = builder.create(loc, unbroadcastedVectorType, + readOp.getMask(), load, fill); + // Insert a broadcasting op if required. + if (requiresBroadcasting) { + res = builder.create(loc, readOp.getVectorType(), res); + } + return res; +} + +static constexpr char kTransferReadNeedsMask[] = + "amdgpu.buffer_transfer_read_needs_mask"; + namespace { struct TransferReadLowering final : OpRewritePattern { @@ -108,6 +140,8 @@ struct TransferReadLowering final : OpRewritePattern { LogicalResult matchAndRewrite(vector::TransferReadOp readOp, PatternRewriter &rewriter) const override { + if (readOp->hasAttr(kTransferReadNeedsMask)) + return failure(); bool requiresBroadcasting = false; VectorType unbroadcastedVectorType; @@ -117,20 +151,115 @@ struct TransferReadLowering final : OpRewritePattern { } Location loc = readOp.getLoc(); - Value fill = rewriter.create(loc, unbroadcastedVectorType, - readOp.getPadding()); - Value load = rewriter.create( - loc, unbroadcastedVectorType, readOp.getSource(), readOp.getIndices()); - Value res = rewriter.create(loc, unbroadcastedVectorType, - readOp.getMask(), load, fill); - - // Insert a broadcasting op if required. - if (requiresBroadcasting) { - res = rewriter.create(loc, readOp.getVectorType(), - res); + Value src = readOp.getSource(); + + VectorType vectorType = readOp.getVectorType(); + int64_t vectorSize = vectorType.getNumElements(); + int64_t elementBitWidth = vectorType.getElementTypeBitWidth(); + SmallVector indices = readOp.getIndices(); + + auto stridedMetadata = + rewriter.create(loc, src); + SmallVector strides = + stridedMetadata.getConstifiedMixedStrides(); + SmallVector sizes = stridedMetadata.getConstifiedMixedSizes(); + OpFoldResult offset = stridedMetadata.getConstifiedMixedOffset(); + OpFoldResult linearizedIndices; + std::tie(std::ignore, linearizedIndices) = + memref::getLinearizedMemRefOffsetAndSize(rewriter, loc, elementBitWidth, + elementBitWidth, offset, sizes, + strides, indices); + + // TODO(jerryyin): Fix the getLinearizedMemRefOffsetAndSize() function + // Note below doesn't give the correct result for the linearized size. + // Value totalSize = getValueOrCreateConstantIndexOp( + // rewriter, loc, linearizedInfo.linearizedSize); + // It computes the multiplied sizes of all dimensions instead of taking + // the maximum of each dimension size * stride. + SmallVector productExpressions; + SmallVector productResults; + unsigned sourceRank = cast(src.getType()).getRank(); + + SmallVector symbols(2 * sourceRank); + SmallVector offsetValues; + bindSymbolsList(rewriter.getContext(), MutableArrayRef{symbols}); + + size_t symbolIndex = 0; + for (size_t i = 0; i < sourceRank; ++i) { + AffineExpr strideExpr, sizeExpr; + OpFoldResult stride = strides[i]; + OpFoldResult size = sizes[i]; + if (auto constantStride = getConstantIntValue(stride)) { + strideExpr = rewriter.getAffineConstantExpr(*constantStride); + } else { + strideExpr = symbols[symbolIndex++]; + offsetValues.push_back( + getValueOrCreateConstantIndexOp(rewriter, loc, stride)); + } + + if (auto constantSize = getConstantIntValue(size)) { + sizeExpr = rewriter.getAffineConstantExpr(*constantSize); + } else { + sizeExpr = symbols[symbolIndex++]; + offsetValues.push_back( + getValueOrCreateConstantIndexOp(rewriter, loc, size)); + } + + productExpressions.push_back(strideExpr * sizeExpr); } - rewriter.replaceOp(readOp, res); + AffineMap maxMap = AffineMap::get( + /*dimCount=*/0, /*symbolCount=*/symbolIndex, productExpressions, + rewriter.getContext()); + Value totalSize = + rewriter.create(loc, maxMap, offsetValues); + + // delta = bufferSize - linearizedOffset + Value vectorSizeOffset = + rewriter.create(loc, vectorSize); + Value linearIndex = + getValueOrCreateConstantIndexOp(rewriter, loc, linearizedIndices); + Value delta = rewriter.create(loc, totalSize, linearIndex); + + // 1) check if delta < vectorSize + Value isOutofBounds = rewriter.create( + loc, arith::CmpIPredicate::ult, delta, vectorSizeOffset); + + // 2) check if (detla_bytes % (32 / elementBitwidth) != 0) + Value deltaBytes = rewriter.create( + loc, delta, + rewriter.create(loc, elementBitWidth / 8)); + Value elementsPerWord = rewriter.create( + loc, llvm::divideCeil(32, elementBitWidth)); + Value isNotWordAligned = rewriter.create( + loc, arith::CmpIPredicate::ne, + rewriter.create(loc, deltaBytes, elementsPerWord), + rewriter.create(loc, 0)); + + // We take the fallback of transfer_read default lowering only it is both + // out-of-bounds and not word aligned. The fallback ensures correct results + // when loading at the boundary of the buffer since buffer load returns + // inconsistent zeros for the whole word when boundary is crossed. + Value ifCondition = + rewriter.create(loc, isOutofBounds, isNotWordAligned); + + auto thenBuilder = [&](OpBuilder &builder, Location loc) { + Operation *read = builder.clone(*readOp.getOperation()); + read->setAttr(kTransferReadNeedsMask, builder.getUnitAttr()); + Value readResult = read->getResult(0); + builder.create(loc, readResult); + }; + + auto elseBuilder = [&](OpBuilder &builder, Location loc) { + Value res = createVectorLoadForMaskedLoad( + builder, loc, readOp, requiresBroadcasting, unbroadcastedVectorType); + rewriter.create(loc, res); + }; + + auto ifOp = + rewriter.create(loc, ifCondition, thenBuilder, elseBuilder); + + rewriter.replaceOp(readOp, ifOp); return success(); } @@ -149,6 +278,8 @@ struct AmdgpuTransferReadToLoadPass final void runOnOperation() override { RewritePatternSet patterns(&getContext()); populateAmdgpuTransferReadToLoadPatterns(patterns); - walkAndApplyPatterns(getOperation(), std::move(patterns)); + if (failed(applyPatternsGreedily(getOperation(), std::move(patterns)))) { + return signalPassFailure(); + } } }; diff --git a/mlir/test/Dialect/AMDGPU/transfer-read-to-load.mlir b/mlir/test/Dialect/AMDGPU/transfer-read-to-load.mlir index 3e1283579f2b1..d0805b6b8a973 100644 --- a/mlir/test/Dialect/AMDGPU/transfer-read-to-load.mlir +++ b/mlir/test/Dialect/AMDGPU/transfer-read-to-load.mlir @@ -9,11 +9,71 @@ func.func @transfer_to_maskedload_fatrawbuffer(%mem : memref<8x8xf32, #amdgpu.ad %res = vector.transfer_read %mem[%idx, %idx], %cf0, %mask {in_bounds = [true]} : memref<8x8xf32, #amdgpu.address_space>, vector<4xf32> return %res : vector<4xf32> } -// CHECK: %[[CST:.*]] = arith.constant 0.0 -// CHECK: %[[SPLAT:.*]] = vector.splat %[[CST]] + +// CHECK: %[[FALSE:.*]] = arith.constant false +// CHECK: %[[IF:.*]] = scf.if %[[FALSE]] -> (vector<4xf32>) { +// CHECK: vector.transfer_read %[[ARG0]][%[[ARG1]], %[[ARG1]]] + +// CHECK: } else { // CHECK: %[[LOAD:.*]] = vector.load %arg0[%arg1, %arg1] -// CHECK: %[[SELECT:.*]] = arith.select %arg2, %[[LOAD]], %[[SPLAT]] -// CHECK: return %[[SELECT]] : vector<4xf32> +// CHECK: %[[SELECT:.*]] = arith.select %[[ARG2]], %[[LOAD]] + +// CHECK: return %[[IF]] : vector<4xf32> + +// ----- + +// CHECK: #map = affine_map<()[s0, s1] -> (s0 * 8 + s1)> +// CHECK-LABEL: func @transfer_to_maskedload_fatrawbuffer_f16( +// CHECK-SAME: %[[ARG0:.+]]: memref<8x8xf16, #amdgpu.address_space>, +// CHECK-SAME: %[[ARG1:.+]]: index, %[[ARG2:.+]]: index, +// CHECK-SAME: %[[ARG3:.+]]: vector<4xi1>) +func.func @transfer_to_maskedload_fatrawbuffer_f16(%mem : memref<8x8xf16, #amdgpu.address_space>, %idx0 : index, %idx1 : index, %mask : vector<4xi1>) -> vector<4xf16> { + %cf0 = arith.constant 0.0 : f16 + %res = vector.transfer_read %mem[%idx0, %idx1], %cf0, %mask {in_bounds = [true]} : memref<8x8xf16, #amdgpu.address_space>, vector<4xf16> + return %res : vector<4xf16> +} +// CHECK-DAG: %[[C0:.*]] = arith.constant 0 +// CHECK-DAG: %[[SIZE:.*]] = arith.constant 64 +// CHECK-DAG: %[[BYTES:.*]] = arith.constant 2 +// CHECK-DAG: %[[VECTORSIZE:.*]] = arith.constant 4 + +// CHECK: %[[LINEAR:.*]] = affine.apply #map()[%[[ARG1]], %[[ARG2]]] +// CHECK: %[[DELTA:.*]] = arith.subi %[[SIZE]], %[[LINEAR]] +// CHECK: %[[COND1:.*]] = arith.cmpi ult, %[[DELTA]], %[[VECTORSIZE]] + +// CHECK: %[[DELTABYTES:.*]] = arith.muli %[[DELTA]], %[[BYTES]] +// CHECK: %[[REM:.*]] = arith.remui %[[DELTABYTES]], %[[BYTES]] +// CHECK: %[[COND2:.*]] = arith.cmpi ne, %[[REM]], %[[C0]] + +// CHECK: %[[COND:.*]] = arith.andi %[[COND1]], %[[COND2]] +// CHECK: %[[IF:.*]] = scf.if %[[COND]] -> (vector<4xf16>) { +// CHECK: vector.transfer_read %[[ARG0]][%[[ARG1]], %[[ARG2]]] +// CHECK: } else { +// CHECK: %[[LOAD:.*]] = vector.load %[[ARG0]][%[[ARG1]], %[[ARG2]]] +// CHECK: return %[[IF]] : vector<4xf16> + +// ----- + +// CHECK: #map = affine_map<()[s0, s1, s2] -> (s0 * s1 + s2)> +// CHECK: #map1 = affine_map<()[s0, s1, s2] -> (s0 * s1, s2)> +// CHECK-LABEL: func @transfer_to_maskedload_fatrawbuffer_dynamic_i8( +// CHECK-SAME: %[[ARG0:.*]]: memref> +// CHECK-SAME: %[[ARG1:.*]]: index, %[[ARG2:.*]]: index +// CHECK-SAME: %[[ARG3:.*]]: vector<4xi1> +func.func @transfer_to_maskedload_fatrawbuffer_dynamic_i8(%mem : memref>, %idx0 : index, %idx1 : index, %mask : vector<4xi1>) -> vector<4xi8> { + %cf0 = arith.constant 0 : i8 + %res = vector.transfer_read %mem[%idx0, %idx1], %cf0, %mask {in_bounds = [true]} : memref>, vector<4xi8> + return %res : vector<4xi8> +} + +// CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<4xi8> +// CHECK: %[[C0:.*]] = arith.constant 0 : index +// CHECK: %[[C4:.*]] = arith.constant 4 : index +// CHECK: %[[BASE:.*]], %[[OFFSET:.*]], %[[SIZES:.*]]:2, %[[STRIDES:.*]]:2 = memref.extract_strided_metadata %[[ARG0]] +// CHECK: %[[LINEAR:.*]] = affine.apply #map()[%[[ARG1]], %[[STRIDES]]#0, %[[ARG2]]] +// CHECK: %[[SIZE:.*]] = affine.max #map1()[%[[STRIDES]]#0, %[[SIZES]]#0, %[[SIZES]]#1] +// CHECK: %[[IF:.*]] = scf.if +// CHECK: return // ----- @@ -26,8 +86,8 @@ func.func @transfer_to_maskedload_regular(%mem : memref<8x8xf32>, %idx : index, %res = vector.transfer_read %mem[%idx, %idx], %cf0, %mask {in_bounds = [true]} : memref<8x8xf32>, vector<4xf32> return %res : vector<4xf32> } -// CHECK: %[[CST:.*]] = arith.constant 0.0 -// CHECK: %[[RES:.*]] = vector.transfer_read %arg0[%arg1, %arg1], %[[CST]], %arg2 {in_bounds = [true]} : memref<8x8xf32>, vector<4xf32> +// CHECK: %[[CST:.*]] = arith.constant 0.000000e+00 +// CHECK: %[[RES:.*]] = vector.transfer_read %[[ARG0]][%[[ARG1]], %[[ARG1]]], %[[CST]], %[[ARG2]] // CHECK: return %[[RES]] : vector<4xf32> // ----- @@ -41,8 +101,8 @@ func.func @transfer_to_maskedload_addrspace(%mem : memref<8x8xf32, #gpu.address_ %res = vector.transfer_read %mem[%idx, %idx], %cf0, %mask {in_bounds = [true]} : memref<8x8xf32, #gpu.address_space>, vector<4xf32> return %res : vector<4xf32> } -// CHECK: %[[CST:.*]] = arith.constant 0.0 -// CHECK: %[[RES:.*]] = vector.transfer_read %arg0[%arg1, %arg1], %[[CST]], %arg2 {in_bounds = [true]} : memref<8x8xf32, #gpu.address_space>, vector<4xf32> +// CHECK: %[[CST:.*]] = arith.constant 0.000000e+00 +// CHECK: %[[RES:.*]] = vector.transfer_read %[[ARG0]][%[[ARG1]], %[[ARG1]]], %[[CST]], %[[ARG2]] // CHECK: return %[[RES]] : vector<4xf32> // ----- @@ -59,12 +119,12 @@ func.func @transfer_broadcasting(%mem : memref<8x8xf32, #amdgpu.address_space>, vector<4xf32> return %res : vector<4xf32> } -// CHECK: %[[CST:.*]] = arith.constant 0.0 -// CHECK: %[[SPLAT:.*]] = vector.splat %[[CST]] +// CHECK: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<1xf32> +// CHECK: %[[FALSE:.*]] = arith.constant false +// CHECK: %[[IF:.*]] = scf.if %[[FALSE]] -> (vector<4xf32>) { // CHECK: %[[LOAD:.*]] = vector.load %arg0[%arg1, %arg1] -// CHECK: %[[SELECT:.*]] = arith.select %arg2, %[[LOAD]], %[[SPLAT]] +// CHECK: %[[SELECT:.*]] = arith.select %arg2, %[[LOAD]], %[[CST]] // CHECK: %[[BROADCAST:.*]] = vector.broadcast %[[SELECT]] : vector<1xf32> to vector<4xf32> -// CHECK: return %[[BROADCAST]] : vector<4xf32> // ----- @@ -79,8 +139,8 @@ func.func @transfer_scalar(%mem : memref<8x8xf32, #amdgpu.address_space>, vector<1xf32> return %res : vector<1xf32> } -// CHECK: %[[CST:.*]] = arith.constant 0.0 -// CHECK: %[[SPLAT:.*]] = vector.splat %[[CST]] -// CHECK: %[[LOAD:.*]] = vector.load %arg0[%arg1, %arg1] -// CHECK: %[[SELECT:.*]] = arith.select %arg2, %[[LOAD]], %[[SPLAT]] -// CHECK: return %[[SELECT]] : vector<1xf32> +// CHECK: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<1xf32> +// CHECK: %[[FALSE:.*]] = arith.constant false +// CHECK: %[[IF:.*]] = scf.if %[[FALSE]] -> (vector<1xf32>) { +// CHECK: %[[LOAD:.*]] = vector.load %[[ARG0]][%[[ARG1]], %[[ARG1]]] +// CHECK: %[[SELECT:.*]] = arith.select %arg2, %[[LOAD]], %[[CST]] diff --git a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel index 07c522a237a8a..10503fe1d123b 100644 --- a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel @@ -1569,6 +1569,7 @@ cc_library( ":IR", ":MemRefDialect", ":Pass", + ":SCFDialect", ":SideEffectInterfaces", ":Support", ":TransformUtils", From f83c5fe01fbee0f53ecf69d887e7a7b054f2a9ae Mon Sep 17 00:00:00 2001 From: Mircea Trofin Date: Tue, 15 Apr 2025 13:55:40 -0700 Subject: [PATCH 26/63] [nfc] Expose `canReturn` from FunctionAttrs (#135650) This is a fairly light-weight traversal and is needed in instrumentation. No need to run the whole `FunctionAttrs` pass at this stage. To avoid layering issues, this patch factors `canRun` and related under Analysis/CFG. --- llvm/include/llvm/Analysis/CFG.h | 1 + llvm/lib/Analysis/CFG.cpp | 34 ++++++++++++++++++++++ llvm/lib/Transforms/IPO/FunctionAttrs.cpp | 35 ----------------------- 3 files changed, 35 insertions(+), 35 deletions(-) diff --git a/llvm/include/llvm/Analysis/CFG.h b/llvm/include/llvm/Analysis/CFG.h index 23bc10a4a9d1b..8451e88146d7c 100644 --- a/llvm/include/llvm/Analysis/CFG.h +++ b/llvm/include/llvm/Analysis/CFG.h @@ -174,6 +174,7 @@ bool containsIrreducibleCFG(RPOTraversalT &RPOTraversal, const LoopInfoT &LI) { return false; } +bool canReturn(const Function &F); } // End llvm namespace #endif diff --git a/llvm/lib/Analysis/CFG.cpp b/llvm/lib/Analysis/CFG.cpp index 841b835052380..8ced4a901557d 100644 --- a/llvm/lib/Analysis/CFG.cpp +++ b/llvm/lib/Analysis/CFG.cpp @@ -322,3 +322,37 @@ bool llvm::isPotentiallyReachable( return isPotentiallyReachable( A->getParent(), B->getParent(), ExclusionSet, DT, LI); } + +static bool instructionDoesNotReturn(const Instruction &I) { + if (auto *CB = dyn_cast(&I)) + return CB->hasFnAttr(Attribute::NoReturn); + return false; +} + +// A basic block can only return if it terminates with a ReturnInst and does not +// contain calls to noreturn functions. +static bool basicBlockCanReturn(const BasicBlock &BB) { + if (!isa(BB.getTerminator())) + return false; + return none_of(BB, instructionDoesNotReturn); +} + +// FIXME: this doesn't handle recursion. +bool llvm::canReturn(const Function &F) { + SmallVector Worklist; + SmallPtrSet Visited; + + Visited.insert(&F.front()); + Worklist.push_back(&F.front()); + + do { + const BasicBlock *BB = Worklist.pop_back_val(); + if (basicBlockCanReturn(*BB)) + return true; + for (const BasicBlock *Succ : successors(BB)) + if (Visited.insert(Succ).second) + Worklist.push_back(Succ); + } while (!Worklist.empty()); + + return false; +} diff --git a/llvm/lib/Transforms/IPO/FunctionAttrs.cpp b/llvm/lib/Transforms/IPO/FunctionAttrs.cpp index ef7989507c89f..bbfed2ac2c090 100644 --- a/llvm/lib/Transforms/IPO/FunctionAttrs.cpp +++ b/llvm/lib/Transforms/IPO/FunctionAttrs.cpp @@ -2090,41 +2090,6 @@ static void addNoRecurseAttrs(const SCCNodeSet &SCCNodes, Changed.insert(F); } -static bool instructionDoesNotReturn(Instruction &I) { - if (auto *CB = dyn_cast(&I)) - return CB->hasFnAttr(Attribute::NoReturn); - return false; -} - -// A basic block can only return if it terminates with a ReturnInst and does not -// contain calls to noreturn functions. -static bool basicBlockCanReturn(BasicBlock &BB) { - if (!isa(BB.getTerminator())) - return false; - return none_of(BB, instructionDoesNotReturn); -} - -// FIXME: this doesn't handle recursion. -static bool canReturn(Function &F) { - SmallVector Worklist; - SmallPtrSet Visited; - - Visited.insert(&F.front()); - Worklist.push_back(&F.front()); - - do { - BasicBlock *BB = Worklist.pop_back_val(); - if (basicBlockCanReturn(*BB)) - return true; - for (BasicBlock *Succ : successors(BB)) - if (Visited.insert(Succ).second) - Worklist.push_back(Succ); - } while (!Worklist.empty()); - - return false; -} - - // Set the noreturn function attribute if possible. static void addNoReturnAttrs(const SCCNodeSet &SCCNodes, SmallSet &Changed) { From 12697c5516f8a9e4407e01d99324ce6958910184 Mon Sep 17 00:00:00 2001 From: Craig Topper Date: Tue, 15 Apr 2025 13:56:07 -0700 Subject: [PATCH 27/63] [LegalizeTypes] Check getTypeAction before calling GetScalarizedVector. (#135838) Use getTypeAction instead of trying to guess how a type will be legalized. On AArch64, v1f16 is scalarized but v1f16 is widened. Fixes #135776 --- .../SelectionDAG/LegalizeVectorTypes.cpp | 4 +--- llvm/test/CodeGen/AArch64/pr135776.ll | 17 +++++++++++++++++ 2 files changed, 18 insertions(+), 3 deletions(-) create mode 100644 llvm/test/CodeGen/AArch64/pr135776.ll diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp index f934d8b37561e..a01e1cff74564 100644 --- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp +++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp @@ -400,9 +400,7 @@ SDValue DAGTypeLegalizer::ScalarizeVecRes_MERGE_VALUES(SDNode *N, SDValue DAGTypeLegalizer::ScalarizeVecRes_BITCAST(SDNode *N) { SDValue Op = N->getOperand(0); - if (Op.getValueType().isVector() - && Op.getValueType().getVectorNumElements() == 1 - && !isSimpleLegalType(Op.getValueType())) + if (getTypeAction(Op.getValueType()) == TargetLowering::TypeScalarizeVector) Op = GetScalarizedVector(Op); EVT NewVT = N->getValueType(0).getVectorElementType(); return DAG.getNode(ISD::BITCAST, SDLoc(N), diff --git a/llvm/test/CodeGen/AArch64/pr135776.ll b/llvm/test/CodeGen/AArch64/pr135776.ll new file mode 100644 index 0000000000000..6f026234664fe --- /dev/null +++ b/llvm/test/CodeGen/AArch64/pr135776.ll @@ -0,0 +1,17 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 + +; RUN: llc < %s -mtriple=aarch64-linux-gnu | FileCheck %s + +define i32 @bitcast_failure(ptr %0, <1 x i16> %1) { +; CHECK-LABEL: bitcast_failure: +; CHECK: // %bb.0: +; CHECK-NEXT: mov x8, x0 +; CHECK-NEXT: mov w0, wzr +; CHECK-NEXT: // kill: def $d0 killed $d0 def $q0 +; CHECK-NEXT: str h0, [x8] +; CHECK-NEXT: ret + %3 = bitcast <1 x i16> %1 to <1 x half> + %4 = extractelement <1 x half> %3, i64 0 + store half %4, ptr %0, align 2 + ret i32 0 +} From ddb12674300eb1af5e6945b5447e7bff7cff4cd8 Mon Sep 17 00:00:00 2001 From: Alexey Bataev Date: Tue, 15 Apr 2025 13:53:16 -0700 Subject: [PATCH 28/63] [SLP]Insert vector instruction after landingpad If the node must be emitted in the landingpad block, need to insert the instructions after the landingpad instruction to avoid a crash. Fixes #135781 --- .../Transforms/Vectorize/SLPVectorizer.cpp | 6 +- .../X86/landing-pad-for-split-node.ll | 77 +++++++++++++++++++ 2 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/landing-pad-for-split-node.ll diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp index b174f0f03fca6..b48674f6993e3 100644 --- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp +++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp @@ -16010,8 +16010,12 @@ void BoUpSLP::setInsertPointAfterBundle(const TreeEntry *E) { BasicBlock::iterator LastInstIt = LastInst->getIterator(); // If the instruction is PHI, set the insert point after all the PHIs. bool IsPHI = isa(LastInst); - if (IsPHI) + if (IsPHI) { LastInstIt = LastInst->getParent()->getFirstNonPHIIt(); + if (LastInstIt != LastInst->getParent()->end() && + LastInstIt->getParent()->isLandingPad()) + LastInstIt = std::next(LastInstIt); + } if (IsPHI || (!E->isGather() && E->State != TreeEntry::SplitVectorize && doesNotNeedToSchedule(E->Scalars)) || diff --git a/llvm/test/Transforms/SLPVectorizer/X86/landing-pad-for-split-node.ll b/llvm/test/Transforms/SLPVectorizer/X86/landing-pad-for-split-node.ll new file mode 100644 index 0000000000000..d6552adbd4abf --- /dev/null +++ b/llvm/test/Transforms/SLPVectorizer/X86/landing-pad-for-split-node.ll @@ -0,0 +1,77 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5 +; RUN: opt -S --passes=slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -slp-threshold=-9999 < %s | FileCheck %s + +define void @test(i32 %arg) personality ptr null { +; CHECK-LABEL: define void @test( +; CHECK-SAME: i32 [[ARG:%.*]]) personality ptr null { +; CHECK-NEXT: [[BB:.*]]: +; CHECK-NEXT: [[LOAD:%.*]] = load i32, ptr addrspace(3) null, align 4 +; CHECK-NEXT: [[LOAD1:%.*]] = load i32, ptr addrspace(3) null, align 4 +; CHECK-NEXT: [[LOAD2:%.*]] = load i32, ptr addrspace(3) null, align 4 +; CHECK-NEXT: [[LOAD3:%.*]] = load i32, ptr addrspace(3) null, align 4 +; CHECK-NEXT: [[INVOKE:%.*]] = invoke i32 null(ptr addrspace(1) null, i32 0) +; CHECK-NEXT: to label %[[BB4:.*]] unwind label %[[BB5:.*]] +; CHECK: [[BB4]]: +; CHECK-NEXT: ret void +; CHECK: [[BB5]]: +; CHECK-NEXT: [[PHI:%.*]] = phi i32 [ 0, %[[BB]] ] +; CHECK-NEXT: [[PHI6:%.*]] = phi i32 [ 0, %[[BB]] ] +; CHECK-NEXT: [[PHI7:%.*]] = phi i32 [ 0, %[[BB]] ] +; CHECK-NEXT: [[PHI8:%.*]] = phi i32 [ 0, %[[BB]] ] +; CHECK-NEXT: [[LANDINGPAD:%.*]] = landingpad { ptr, i32 } +; CHECK-NEXT: cleanup +; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[LOAD]], i32 0 +; CHECK-NEXT: [[TMP1:%.*]] = insertelement <4 x i32> [[TMP0]], i32 [[LOAD1]], i32 1 +; CHECK-NEXT: [[TMP2:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[LOAD3]], i32 2 +; CHECK-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> [[TMP2]], i32 [[LOAD2]], i32 3 +; CHECK-NEXT: [[TMP4:%.*]] = insertelement <4 x i32> poison, i32 [[PHI]], i32 0 +; CHECK-NEXT: [[TMP5:%.*]] = insertelement <4 x i32> [[TMP4]], i32 [[PHI8]], i32 1 +; CHECK-NEXT: [[TMP6:%.*]] = insertelement <4 x i32> [[TMP5]], i32 [[PHI6]], i32 2 +; CHECK-NEXT: [[TMP7:%.*]] = insertelement <4 x i32> [[TMP6]], i32 [[PHI7]], i32 3 +; CHECK-NEXT: [[TMP8:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> poison, <8 x i32> +; CHECK-NEXT: [[TMP9:%.*]] = call <8 x i32> @llvm.vector.insert.v8i32.v4i32(<8 x i32> [[TMP8]], <4 x i32> [[TMP7]], i64 4) +; CHECK-NEXT: br label %[[BB11:.*]] +; CHECK: [[BB9:.*]]: +; CHECK-NEXT: [[LANDINGPAD10:%.*]] = landingpad { ptr, i32 } +; CHECK-NEXT: cleanup +; CHECK-NEXT: br label %[[BB11]] +; CHECK: [[BB11]]: +; CHECK-NEXT: [[TMP10:%.*]] = phi <8 x i32> [ poison, %[[BB9]] ], [ [[TMP9]], %[[BB5]] ] +; CHECK-NEXT: ret void +; +bb: + %load = load i32, ptr addrspace(3) null, align 4 + %load1 = load i32, ptr addrspace(3) null, align 4 + %load2 = load i32, ptr addrspace(3) null, align 4 + %load3 = load i32, ptr addrspace(3) null, align 4 + %invoke = invoke i32 null(ptr addrspace(1) null, i32 0) + to label %bb4 unwind label %bb5 + +bb4: ; preds = %bb + ret void + +bb5: ; preds = %bb + %phi = phi i32 [ 0, %bb ] + %phi6 = phi i32 [ 0, %bb ] + %phi7 = phi i32 [ 0, %bb ] + %phi8 = phi i32 [ 0, %bb ] + %landingpad = landingpad { ptr, i32 } + cleanup + br label %bb11 + +bb9: ; No predecessors! + %landingpad10 = landingpad { ptr, i32 } + cleanup + br label %bb11 + +bb11: ; preds = %bb9, %bb5 + %phi12 = phi i32 [ 0, %bb9 ], [ %phi, %bb5 ] + %phi13 = phi i32 [ 0, %bb9 ], [ %phi8, %bb5 ] + %phi14 = phi i32 [ 0, %bb9 ], [ %phi6, %bb5 ] + %phi15 = phi i32 [ %arg, %bb9 ], [ %phi7, %bb5 ] + %phi16 = phi i32 [ 0, %bb9 ], [ %load, %bb5 ] + %phi17 = phi i32 [ 0, %bb9 ], [ %load1, %bb5 ] + %phi18 = phi i32 [ %arg, %bb9 ], [ %load2, %bb5 ] + %phi19 = phi i32 [ 0, %bb9 ], [ %load3, %bb5 ] + ret void +} From 0f3e460e06e03ce37445546457a16d6f1eee1e21 Mon Sep 17 00:00:00 2001 From: MaheshRavishankar <1663364+MaheshRavishankar@users.noreply.github.com> Date: Tue, 15 Apr 2025 14:10:18 -0700 Subject: [PATCH 29/63] [mlir][Tensor] Generalize the pattern to swap `tensor.collapse_shape` -> `tensor.expand_shape`. (#133819) The current patterns compared the reassocation indices for the two ops and failed if neither of them were of size 1. This patch relaxes this restriction by handling a new case where the reassociation indices might be of the same size. Also generalizes to cases where when generating the swapped `tensor.expand_shape` -> `tensor.collapse_shape` if one of them is degenerate, those are not generated. Signed-off-by: MaheshRavishankar --- .../Tensor/Transforms/ReshapePatterns.cpp | 112 ++++++++++++++---- mlir/test/Dialect/Tensor/bubble-reshapes.mlir | 63 +++++++++- 2 files changed, 149 insertions(+), 26 deletions(-) diff --git a/mlir/lib/Dialect/Tensor/Transforms/ReshapePatterns.cpp b/mlir/lib/Dialect/Tensor/Transforms/ReshapePatterns.cpp index eed44e60d6591..a3de7f9b44ae6 100644 --- a/mlir/lib/Dialect/Tensor/Transforms/ReshapePatterns.cpp +++ b/mlir/lib/Dialect/Tensor/Transforms/ReshapePatterns.cpp @@ -167,10 +167,39 @@ struct BubbleUpExpandThroughParallelCollapse return failure(); } - // Reshapes are parallel to each other if none of the reassociation indices - // have greater than 1 index for both reshapes. + // Reshapes are parallel to each other (by construction the number of + // reassociations specified in the collapse and expand are the same), if at + // any position + // 1. either the reassociation indices are of the same size, or + // 2. either the reassociation in the collapse or the expand is of size 1. + ArrayRef staticSourceSize = collapseOp.getSrcType().getShape(); + ArrayRef staticResultSize = expandOp.getStaticOutputShape(); for (auto [expandReassociation, collapseReassociation] : llvm::zip_equal(expandReInds, collapseReInds)) { + if (collapseReassociation.size() == expandReassociation.size()) { + // Even if the reassociations are the same, the collapse/expand should + // result in the same dimensions. i.e 4x8x2 into 64 should be expanded + // into 4x8x2 again. In presense of dynamic dimensions one can only + // verify "equality" when there is only one dynamic dimension present, + // and all other static dimensions are equal. + ArrayRef collapsedStaticShapes = staticSourceSize.slice( + collapseReassociation.front(), collapseReassociation.size()); + int64_t numCollapsedDynamic = + llvm::count_if(collapsedStaticShapes, + [](int64_t d) { return ShapedType::isDynamic(d); }); + ArrayRef expandedStaticShapes = staticResultSize.slice( + expandReassociation.front(), expandReassociation.size()); + int64_t numExpandedDynamic = + llvm::count_if(expandedStaticShapes, + [](int64_t d) { return ShapedType::isDynamic(d); }); + if (numCollapsedDynamic > 1 || numExpandedDynamic > 1 || + collapsedStaticShapes != expandedStaticShapes) { + return failure(); + } + continue; + } + // If the reassociations are not same, one or the other needs to be of + // size one. if (collapseReassociation.size() != 1 && expandReassociation.size() != 1) return failure(); } @@ -178,33 +207,60 @@ struct BubbleUpExpandThroughParallelCollapse // Compute new reassociation indices and expanded/collaped shapes. SmallVector newExpandReInds, newCollapseReInds; Location loc = expandOp->getLoc(); - SmallVector collapseSizes = + SmallVector sourceSizes = tensor::getMixedSizes(rewriter, loc, collapseOp.getSrc()); - SmallVector expandSizes(getMixedValues( - expandOp.getStaticOutputShape(), expandOp.getOutputShape(), rewriter)); + SmallVector resultSizes = expandOp.getMixedOutputShape(); SmallVector newExpandSizes; - int64_t index = 0, expandIndex = 0, collapseIndex = 0; - for (auto [idx, collapseReassociation] : llvm::enumerate(collapseReInds)) { + + int64_t newExpandIndex = 0, newCollapseIndex = 0, sourceSizeIndex = 0, + resultSizeIndex = 0; + + for (size_t idx = 0, idxEnd = collapseReInds.size(); idx < idxEnd; idx++) { + auto &collapseReassociation = collapseReInds[idx]; + auto &expandReassociation = expandReInds[idx]; + + // Case 1. The reassociations are same in the collapse producer + // and expand consumer. In the swapped expand, each of the final + // dimensions are kept as is in the expand and the collapse. So, + // for every element in the `ReassocationIndices` vector add a new + // `ReassociationIndices` vector for the swapped expand and collapse + // (of size 1). + if (collapseReassociation.size() == expandReassociation.size()) { + for (size_t i = 0; i < collapseReassociation.size(); ++i) { + newCollapseReInds.push_back({newCollapseIndex++}); + newExpandReInds.push_back({newExpandIndex++}); + newExpandSizes.push_back(resultSizes[resultSizeIndex++]); + sourceSizeIndex++; + } + continue; + } + + // Case 2. The `ReassociationIndices` in the collapse is of size > 1 (and + // in the expand is of size == 1). In this case, the original dimensions + // are preserved on expansion and collapsed subsequently. if (collapseReassociation.size() != 1) { ReassociationIndices newCollapseReassociation; for (size_t i = 0; i < collapseReassociation.size(); ++i) { - newCollapseReassociation.push_back(index); - newExpandReInds.push_back({index++}); - newExpandSizes.push_back(collapseSizes[collapseIndex++]); + newCollapseReassociation.push_back(newCollapseIndex++); + newExpandReInds.push_back({newExpandIndex++}); + newExpandSizes.push_back(sourceSizes[sourceSizeIndex++]); } + resultSizeIndex++; newCollapseReInds.push_back(newCollapseReassociation); - expandIndex++; continue; } + + // Case 3. The `ReassociationIndices` in the expand is of size > 1 (and + // in the collapse is of size == 1). In this case, the expansion happens + // first and the expanded dimensions are preserved on collapse. ReassociationIndices newExpandReassociation; - auto expandReassociation = expandReInds[idx]; for (size_t i = 0; i < expandReassociation.size(); ++i) { - newExpandReassociation.push_back(index); - newCollapseReInds.push_back({index++}); - newExpandSizes.push_back(expandSizes[expandIndex++]); + newExpandReassociation.push_back(newExpandIndex++); + newCollapseReInds.push_back({newCollapseIndex++}); + newExpandSizes.push_back(resultSizes[resultSizeIndex++]); } newExpandReInds.push_back(newExpandReassociation); - collapseIndex++; + sourceSizeIndex++; } // Swap reshape order. @@ -212,11 +268,25 @@ struct BubbleUpExpandThroughParallelCollapse SmallVector staticSizes; dispatchIndexOpFoldResults(newExpandSizes, dynamicSizes, staticSizes); auto expandResultType = expandOp.getResultType().clone(staticSizes); - auto newExpand = rewriter.create( - loc, expandResultType, collapseOp.getSrc(), newExpandReInds, - newExpandSizes); - rewriter.replaceOpWithNewOp( - expandOp, newExpand.getResult(), newCollapseReInds); + Value newCollapseSrc = collapseOp.getSrc(); + // If the number of reassociation indices in the new `expand_shape` op + // matches the number of dimensions of the result, then the expand_shape + // is a no-op. + if (newExpandReInds.size() != newExpandSizes.size()) { + newCollapseSrc = rewriter.create( + loc, expandResultType, newCollapseSrc, newExpandReInds, + newExpandSizes); + } + + // If the number of reassociation indices in the new `collapse_shape` op + // matches the number of dimensions of the source, then the collapse_shape + // is a no-op. + Value replacement = newCollapseSrc; + if (newCollapseReInds.size() != newExpandSizes.size()) { + replacement = rewriter.create( + loc, newCollapseSrc, newCollapseReInds); + } + rewriter.replaceOp(expandOp, replacement); return success(); } }; diff --git a/mlir/test/Dialect/Tensor/bubble-reshapes.mlir b/mlir/test/Dialect/Tensor/bubble-reshapes.mlir index eeed794884942..81bf8e3f60e2c 100644 --- a/mlir/test/Dialect/Tensor/bubble-reshapes.mlir +++ b/mlir/test/Dialect/Tensor/bubble-reshapes.mlir @@ -48,14 +48,67 @@ func.func @no_bubble_partial_intersecting_reshapes(%arg0: tensor, % // ----- -func.func @no_bubble_0d_tensor_reshapes(%arg0: tensor, %s0: index, %s1: index, %s2: index, %s3: index) -> tensor { - %collapse = tensor.collapse_shape %arg0 [] : tensor into tensor +func.func @no_bubble_0d_tensor_reshapes(%arg0: tensor<1x1xf32>) -> tensor<1x1x1xf32> { + %collapse = tensor.collapse_shape %arg0 [] : tensor<1x1xf32> into tensor %expand = tensor.expand_shape %collapse [] - output_shape [%s0, %s1, %s2, %s3] : tensor into tensor - return %expand : tensor + output_shape [1, 1, 1] : tensor into tensor<1x1x1xf32> + return %expand : tensor<1x1x1xf32> } // CHECK: func @no_bubble_0d_tensor_reshapes -// CHECK-SAME: %[[ARG0:.+]]: tensor +// CHECK-SAME: %[[ARG0:.+]]: tensor<1x1xf32> // CHECK: %[[COLLAPSE:.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}] // CHECK: %[[EXPAND:.+]] = tensor.expand_shape %[[COLLAPSE]] {{\[}}] // CHECK: return %[[EXPAND]] + +// ----- + +// Test the case where the reassocation indices in the collapse and expand +// are of same size. +func.func @bubble_expand_match_non_unit_size_reassocation( + %arg0 : tensor<4x?x4x32x4x?xf16>, %arg1 : index, %arg2 : index) -> tensor<4x?x4x128x?x32xf16> { + %collapsed = tensor.collapse_shape %arg0 [[0, 1, 2], [3, 4], [5]] + : tensor<4x?x4x32x4x?xf16> into tensor + %expanded = tensor.expand_shape %collapsed [[0, 1, 2], [3], [4, 5]] output_shape [4, %arg1, 4, 128, %arg2, 32] + : tensor into tensor<4x?x4x128x?x32xf16> + return %expanded : tensor<4x?x4x128x?x32xf16> +} +// CHECK: func @bubble_expand_match_non_unit_size_reassocation +// CHECK-SAME: %[[ARG0:.+]]: tensor<4x?x4x32x4x?xf16> +// CHECK-SAME: %[[ARG1:[a-zA-z0-9]+]]: index +// CHECK-SAME: %[[ARG2:[a-zA-z0-9]+]]: index +// CHECK: %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] +// CHECK-SAME: {{\[}}[0], [1], [2], [3], [4], [5, 6]{{\]}} +// CHECK-SAME: [4, %[[ARG1]], 4, 32, 4, %[[ARG2]], 32] +// CHECK: %[[COLLAPSED:.+]] = tensor.collapse_shape %[[EXPANDED]] +// CHECK-SAME: {{\[}}[0], [1], [2], [3, 4], [5], [6]{{\]}} +// CHECK: return %[[COLLAPSED]] + +// ----- + +// Test the case where the trailing collapse isnt needed. +func.func @no_collapse_generated( + %arg0 : tensor<4x?x4x128x?xf16>, %arg1 : index, %arg2 : index) -> tensor<4x?x4x128x?x32xf16> { + %collapsed = tensor.collapse_shape %arg0 [[0, 1, 2], [3], [4]] + : tensor<4x?x4x128x?xf16> into tensor + %expanded = tensor.expand_shape %collapsed [[0, 1, 2], [3], [4, 5]] output_shape [4, %arg1, 4, 128, %arg2, 32] + : tensor into tensor<4x?x4x128x?x32xf16> + return %expanded : tensor<4x?x4x128x?x32xf16> +} +// CHECK: func @no_collapse_generated +// CHECK: %[[EXPANDED:.+]] = tensor.expand_shape +// CHECK: return %[[EXPANDED]] + +// ----- + +// Test the case where the leading expand isnt needed. +func.func @no_expand_generated( + %arg0 : tensor<4x?x4x128x?x?x?xf16>, %arg1 : index, %arg2 : index, %arg3 : index) -> tensor<4x?x4x128x?x?xf16> { + %collapsed = tensor.collapse_shape %arg0 [[0, 1, 2], [3], [4], [5, 6]] + : tensor<4x?x4x128x?x?x?xf16> into tensor + %expanded = tensor.expand_shape %collapsed [[0, 1, 2], [3], [4], [5]] output_shape [4, %arg1, 4, 128, %arg2, %arg3] + : tensor into tensor<4x?x4x128x?x?xf16> + return %expanded : tensor<4x?x4x128x?x?xf16> +} +// CHECK: func @no_expand_generated +// CHECK: %[[EXPANDED:.+]] = tensor.collapse_shape +// CHECK: return %[[EXPANDED]] From d30a5b41fe72a1dd83714d3e21fd539b91e63c8c Mon Sep 17 00:00:00 2001 From: Djordje Todorovic Date: Tue, 15 Apr 2025 23:17:03 +0200 Subject: [PATCH 30/63] [RISCV] Fix xmipscmov extension name (#135647) The right name was used in riscv-toolchain-conventions docs. --- clang/test/Driver/print-supported-extensions-riscv.c | 2 +- llvm/docs/RISCVUsage.rst | 2 +- .../Target/RISCV/Disassembler/RISCVDisassembler.cpp | 4 ++-- llvm/lib/Target/RISCV/RISCVFeatures.td | 12 ++++++------ llvm/lib/Target/RISCV/RISCVInstrInfoXMips.td | 4 ++-- llvm/lib/Target/RISCV/RISCVProcessors.td | 2 +- llvm/lib/Target/RISCV/RISCVSubtarget.cpp | 2 +- llvm/test/CodeGen/RISCV/features-info.ll | 2 +- llvm/test/CodeGen/RISCV/select-and.ll | 2 +- llvm/test/CodeGen/RISCV/select-bare.ll | 2 +- llvm/test/CodeGen/RISCV/select-cc.ll | 2 +- llvm/test/CodeGen/RISCV/select-or.ll | 2 +- llvm/test/MC/RISCV/xmips-invalid.s | 4 ++-- llvm/test/MC/RISCV/xmips-valid.s | 6 +++--- llvm/unittests/TargetParser/RISCVISAInfoTest.cpp | 2 +- 15 files changed, 25 insertions(+), 25 deletions(-) diff --git a/clang/test/Driver/print-supported-extensions-riscv.c b/clang/test/Driver/print-supported-extensions-riscv.c index d06cedac5b1eb..39002d7b4780a 100644 --- a/clang/test/Driver/print-supported-extensions-riscv.c +++ b/clang/test/Driver/print-supported-extensions-riscv.c @@ -161,7 +161,7 @@ // CHECK-NEXT: xcvmac 1.0 'XCVmac' (CORE-V Multiply-Accumulate) // CHECK-NEXT: xcvmem 1.0 'XCVmem' (CORE-V Post-incrementing Load & Store) // CHECK-NEXT: xcvsimd 1.0 'XCVsimd' (CORE-V SIMD ALU) -// CHECK-NEXT: xmipscmove 1.0 'XMIPSCMove' (MIPS conditional move instruction(s) (ccmov)) +// CHECK-NEXT: xmipscmov 1.0 'XMIPSCMov' (MIPS conditional move instruction (mips.ccmov)) // CHECK-NEXT: xmipslsp 1.0 'XMIPSLSP' (MIPS optimization for hardware load-store bonding) // CHECK-NEXT: xsfcease 1.0 'XSfcease' (SiFive sf.cease Instruction) // CHECK-NEXT: xsfvcp 1.0 'XSfvcp' (SiFive Custom Vector Coprocessor Interface Instructions) diff --git a/llvm/docs/RISCVUsage.rst b/llvm/docs/RISCVUsage.rst index cda7e5fec8488..137b537f00ea0 100644 --- a/llvm/docs/RISCVUsage.rst +++ b/llvm/docs/RISCVUsage.rst @@ -491,7 +491,7 @@ The current vendor extensions supported are: ``experimental-Xqcisync`` LLVM implements `version 0.2 of the Qualcomm uC Sync Delay extension specification `__ by Qualcomm. All instructions are prefixed with `qc.` as described in the specification. These instructions are only available for riscv32. -``Xmipscmove`` +``Xmipscmov`` LLVM implements conditional move for the `p8700 processor ` by MIPS. ``Xmipslsp`` diff --git a/llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp b/llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp index 366291b53bebb..27809d96b647c 100644 --- a/llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp +++ b/llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp @@ -737,8 +737,8 @@ static constexpr DecoderListEntry DecoderList32[]{ {DecoderTableXSfsystem32, XSfSystemGroup, "SiFive system extensions"}, {DecoderTableXSfcease32, {RISCV::FeatureVendorXSfcease}, "SiFive sf.cease"}, {DecoderTableXmipslsp32, {RISCV::FeatureVendorXMIPSLSP}, "MIPS mips.lsp"}, - {DecoderTableXmipscmove32, - {RISCV::FeatureVendorXMIPSCMove}, + {DecoderTableXmipscmov32, + {RISCV::FeatureVendorXMIPSCMov}, "MIPS mips.ccmov"}, // Standard Extensions {DecoderTableXCV32, XCVFeatureGroup, "CORE-V extensions"}, diff --git a/llvm/lib/Target/RISCV/RISCVFeatures.td b/llvm/lib/Target/RISCV/RISCVFeatures.td index 21bcf343139c2..f51fcf82077f4 100644 --- a/llvm/lib/Target/RISCV/RISCVFeatures.td +++ b/llvm/lib/Target/RISCV/RISCVFeatures.td @@ -1300,12 +1300,12 @@ def HasVendorXCVbi "'XCVbi' (CORE-V Immediate Branching)">; // MIPS Extensions -def FeatureVendorXMIPSCMove - : RISCVExtension<1, 0, "MIPS conditional move instruction(s) (ccmov)">; -def HasVendorXMIPSCMove - : Predicate<"Subtarget->hasVendorXMIPSCMove()">, - AssemblerPredicate<(all_of FeatureVendorXMIPSCMove), - "'Xmipscmove' ('mips.ccmov' instruction)">; +def FeatureVendorXMIPSCMov + : RISCVExtension<1, 0, "MIPS conditional move instruction (mips.ccmov)">; +def HasVendorXMIPSCMov + : Predicate<"Subtarget->hasVendorXMIPSCMov()">, + AssemblerPredicate<(all_of FeatureVendorXMIPSCMov), + "'Xmipscmov' ('mips.ccmov' instruction)">; def UseCCMovInsn : Predicate<"Subtarget->useCCMovInsn()">; def FeatureVendorXMIPSLSP : RISCVExtension<1, 0, "MIPS optimization for hardware load-store bonding">; diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoXMips.td b/llvm/lib/Target/RISCV/RISCVInstrInfoXMips.td index 9be424310d660..ff751994b89b9 100644 --- a/llvm/lib/Target/RISCV/RISCVInstrInfoXMips.td +++ b/llvm/lib/Target/RISCV/RISCVInstrInfoXMips.td @@ -107,8 +107,8 @@ class SWPFormat // MIPS extensions //===----------------------------------------------------------------------===// -let Predicates = [HasVendorXMIPSCMove], hasSideEffects = 0, mayLoad = 0, mayStore = 0, - DecoderNamespace = "Xmipscmove" in { +let Predicates = [HasVendorXMIPSCMov], hasSideEffects = 0, mayLoad = 0, mayStore = 0, + DecoderNamespace = "Xmipscmov" in { def MIPS_CCMOV : RVInstR4<0b11, 0b011, OPC_CUSTOM_0, (outs GPR:$rd), (ins GPR:$rs1, GPR:$rs2, GPR:$rs3), "mips.ccmov", "$rd, $rs2, $rs1, $rs3">, diff --git a/llvm/lib/Target/RISCV/RISCVProcessors.td b/llvm/lib/Target/RISCV/RISCVProcessors.td index 9d48adeec5e86..ece12c48b1cd9 100644 --- a/llvm/lib/Target/RISCV/RISCVProcessors.td +++ b/llvm/lib/Target/RISCV/RISCVProcessors.td @@ -119,7 +119,7 @@ def MIPS_P8700 : RISCVProcessorModel<"mips-p8700", FeatureStdExtZbb, FeatureStdExtZifencei, FeatureStdExtZicsr, - FeatureVendorXMIPSCMove, + FeatureVendorXMIPSCMov, FeatureVendorXMIPSLSP], [TuneMIPSP8700]>; diff --git a/llvm/lib/Target/RISCV/RISCVSubtarget.cpp b/llvm/lib/Target/RISCV/RISCVSubtarget.cpp index 3c996c82fcec4..b3c313f2ed394 100644 --- a/llvm/lib/Target/RISCV/RISCVSubtarget.cpp +++ b/llvm/lib/Target/RISCV/RISCVSubtarget.cpp @@ -253,5 +253,5 @@ bool RISCVSubtarget::useLoadStorePairs() const { } bool RISCVSubtarget::useCCMovInsn() const { - return UseCCMovInsn && HasVendorXMIPSCMove; + return UseCCMovInsn && HasVendorXMIPSCMov; } diff --git a/llvm/test/CodeGen/RISCV/features-info.ll b/llvm/test/CodeGen/RISCV/features-info.ll index 5d2f0881048bd..d377bda059d33 100644 --- a/llvm/test/CodeGen/RISCV/features-info.ll +++ b/llvm/test/CodeGen/RISCV/features-info.ll @@ -174,7 +174,7 @@ ; CHECK-NEXT: xcvmac - 'XCVmac' (CORE-V Multiply-Accumulate). ; CHECK-NEXT: xcvmem - 'XCVmem' (CORE-V Post-incrementing Load & Store). ; CHECK-NEXT: xcvsimd - 'XCVsimd' (CORE-V SIMD ALU). -; CHECK-NEXT: xmipscmove - 'XMIPSCMove' (MIPS conditional move instruction(s) (ccmov)). +; CHECK-NEXT: xmipscmov - 'XMIPSCMov' (MIPS conditional move instruction (mips.ccmov)). ; CHECK-NEXT: xmipslsp - 'XMIPSLSP' (MIPS optimization for hardware load-store bonding). ; CHECK-NEXT: xsfcease - 'XSfcease' (SiFive sf.cease Instruction). ; CHECK-NEXT: xsfvcp - 'XSfvcp' (SiFive Custom Vector Coprocessor Interface Instructions). diff --git a/llvm/test/CodeGen/RISCV/select-and.ll b/llvm/test/CodeGen/RISCV/select-and.ll index f827e840f4a36..2c9d0a8b56425 100644 --- a/llvm/test/CodeGen/RISCV/select-and.ll +++ b/llvm/test/CodeGen/RISCV/select-and.ll @@ -3,7 +3,7 @@ ; RUN: | FileCheck -check-prefix=RV32I %s ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s \ ; RUN: | FileCheck -check-prefix=RV64I %s -; RUN: llc -mtriple=riscv64 -mattr=+xmipscmove -verify-machineinstrs < %s \ +; RUN: llc -mtriple=riscv64 -mattr=+xmipscmov -verify-machineinstrs < %s \ ; RUN: | FileCheck -check-prefix=RV64I-CCMOV %s ;; There are a few different ways to lower (select (and A, B), X, Y). This test diff --git a/llvm/test/CodeGen/RISCV/select-bare.ll b/llvm/test/CodeGen/RISCV/select-bare.ll index c9e108a1ca9d0..fc8eaa480b116 100644 --- a/llvm/test/CodeGen/RISCV/select-bare.ll +++ b/llvm/test/CodeGen/RISCV/select-bare.ll @@ -1,7 +1,7 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py ; RUN: llc -mtriple=riscv32 -verify-machineinstrs < %s \ ; RUN: | FileCheck %s -check-prefix=RV32I -; RUN: llc -mtriple=riscv64 -mattr=+xmipscmove -verify-machineinstrs < %s \ +; RUN: llc -mtriple=riscv64 -mattr=+xmipscmov -verify-machineinstrs < %s \ ; RUN: | FileCheck -check-prefix=RV64I-CCMOV %s define i32 @bare_select(i1 %a, i32 %b, i32 %c) nounwind { diff --git a/llvm/test/CodeGen/RISCV/select-cc.ll b/llvm/test/CodeGen/RISCV/select-cc.ll index 1c2a0cf007d11..e69dc303d85dc 100644 --- a/llvm/test/CodeGen/RISCV/select-cc.ll +++ b/llvm/test/CodeGen/RISCV/select-cc.ll @@ -3,7 +3,7 @@ ; RUN: | FileCheck -check-prefixes=RV32I %s ; RUN: llc -mtriple=riscv64 -disable-block-placement -verify-machineinstrs < %s \ ; RUN: | FileCheck -check-prefixes=RV64I %s -; RUN: llc -mtriple=riscv64 -mattr=+xmipscmove -verify-machineinstrs < %s \ +; RUN: llc -mtriple=riscv64 -mattr=+xmipscmov -verify-machineinstrs < %s \ ; RUN: | FileCheck -check-prefix=RV64I-CCMOV %s define signext i32 @foo(i32 signext %a, ptr %b) nounwind { diff --git a/llvm/test/CodeGen/RISCV/select-or.ll b/llvm/test/CodeGen/RISCV/select-or.ll index 338c7c06c3ab8..091c8b1a11e71 100644 --- a/llvm/test/CodeGen/RISCV/select-or.ll +++ b/llvm/test/CodeGen/RISCV/select-or.ll @@ -3,7 +3,7 @@ ; RUN: | FileCheck -check-prefix=RV32I %s ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s \ ; RUN: | FileCheck -check-prefix=RV64I %s -; RUN: llc -mtriple=riscv64 -mattr=+xmipscmove -verify-machineinstrs < %s \ +; RUN: llc -mtriple=riscv64 -mattr=+xmipscmov -verify-machineinstrs < %s \ ; RUN: | FileCheck -check-prefix=RV64I-CCMOV %s ;; There are a few different ways to lower (select (or A, B), X, Y). This test diff --git a/llvm/test/MC/RISCV/xmips-invalid.s b/llvm/test/MC/RISCV/xmips-invalid.s index a1c1fd0666e0a..b3834e7b3407f 100644 --- a/llvm/test/MC/RISCV/xmips-invalid.s +++ b/llvm/test/MC/RISCV/xmips-invalid.s @@ -1,5 +1,5 @@ # RUN: not llvm-mc -triple=riscv64 < %s 2>&1 | FileCheck %s -check-prefixes=CHECK-FEATURE -# RUN: not llvm-mc -triple=riscv64 -mattr=+xmipslsp,+xmipscmove < %s 2>&1 | FileCheck %s +# RUN: not llvm-mc -triple=riscv64 -mattr=+xmipslsp,+xmipscmov < %s 2>&1 | FileCheck %s mips.ccmov x0, x1, 0x10 # CHECK: error: invalid operand for instruction @@ -8,7 +8,7 @@ mips.ccmov x10 # CHECK: error: too few operands for instruction mips.ccmov s0, s1, s2, s3 -# CHECK-FEATURE: error: instruction requires the following: 'Xmipscmove' ('mips.ccmov' instruction) +# CHECK-FEATURE: error: instruction requires the following: 'Xmipscmov' ('mips.ccmov' instruction) mips.lwp x10, x11 # CHECK: error: too few operands for instruction diff --git a/llvm/test/MC/RISCV/xmips-valid.s b/llvm/test/MC/RISCV/xmips-valid.s index ba256a823f511..9f31e4fa2038c 100644 --- a/llvm/test/MC/RISCV/xmips-valid.s +++ b/llvm/test/MC/RISCV/xmips-valid.s @@ -1,7 +1,7 @@ -# RUN: llvm-mc %s -triple=riscv64 -mattr=+xmipslsp,+xmipscmove -M no-aliases -show-encoding \ +# RUN: llvm-mc %s -triple=riscv64 -mattr=+xmipslsp,+xmipscmov -M no-aliases -show-encoding \ # RUN: | FileCheck -check-prefixes=CHECK-INST,CHECK-ENC %s -# RUN: llvm-mc -filetype=obj -triple=riscv64 -mattr=+xmipslsp,+xmipscmove < %s \ -# RUN: | llvm-objdump --mattr=+xmipslsp,+xmipscmove -M no-aliases -d - \ +# RUN: llvm-mc -filetype=obj -triple=riscv64 -mattr=+xmipslsp,+xmipscmov < %s \ +# RUN: | llvm-objdump --mattr=+xmipslsp,+xmipscmov -M no-aliases -d - \ # RUN: | FileCheck -check-prefix=CHECK-DIS %s # CHECK-INST: mips.ccmov s0, s1, s2, s3 diff --git a/llvm/unittests/TargetParser/RISCVISAInfoTest.cpp b/llvm/unittests/TargetParser/RISCVISAInfoTest.cpp index 2ec27ba9d91b7..ff0a5e64ab3e1 100644 --- a/llvm/unittests/TargetParser/RISCVISAInfoTest.cpp +++ b/llvm/unittests/TargetParser/RISCVISAInfoTest.cpp @@ -1114,7 +1114,7 @@ R"(All available -march extensions for RISC-V xcvmac 1.0 xcvmem 1.0 xcvsimd 1.0 - xmipscmove 1.0 + xmipscmov 1.0 xmipslsp 1.0 xsfcease 1.0 xsfvcp 1.0 From 9c73eba8aa17cb7ca4248ab1c7f67ea7ec9b50b1 Mon Sep 17 00:00:00 2001 From: Aaron Puchert Date: Tue, 15 Apr 2025 23:21:34 +0200 Subject: [PATCH 31/63] Merge similar Clang Thread Safety attributes (#135561) Some of the old lock-based and new capability-based spellings behave basically in the same way, so merging them simplifies the code significantly. There are two minor functional changes: we only warn (instead of an error) when the try_acquire_capability attribute is used on something else than a function. The alternative would have been to produce an error for the old spelling, but we seem to only warn for all function attributes, so this is arguably more consistent. The second change is that we also check the first argument (which is the value returned for a successful try-acquire) for `this`. But from what I can tell, this code is defunct anyway at the moment (see #31414). --- clang/include/clang/Basic/Attr.td | 65 +++++-------------------- clang/lib/AST/ASTImporter.cpp | 28 ----------- clang/lib/Analysis/ThreadSafety.cpp | 60 ++--------------------- clang/lib/Sema/SemaDeclAttr.cpp | 55 --------------------- clang/lib/Sema/SemaDeclCXX.cpp | 13 ++--- clang/test/Sema/attr-capabilities.c | 2 +- clang/unittests/AST/ASTImporterTest.cpp | 36 -------------- 7 files changed, 21 insertions(+), 238 deletions(-) diff --git a/clang/include/clang/Basic/Attr.td b/clang/include/clang/Basic/Attr.td index 9d4900f3029c8..9465451cbfe1f 100644 --- a/clang/include/clang/Basic/Attr.td +++ b/clang/include/clang/Basic/Attr.td @@ -3818,7 +3818,9 @@ def Capability : InheritableAttr { def AssertCapability : InheritableAttr { let Spellings = [Clang<"assert_capability", 0>, - Clang<"assert_shared_capability", 0>]; + Clang<"assert_shared_capability", 0>, + GNU<"assert_exclusive_lock">, + GNU<"assert_shared_lock">]; let Subjects = SubjectList<[Function]>; let LateParsed = LateAttrParseStandard; let TemplateDependent = 1; @@ -3826,7 +3828,8 @@ def AssertCapability : InheritableAttr { let InheritEvenIfAlreadyPresent = 1; let Args = [VariadicExprArgument<"Args">]; let Accessors = [Accessor<"isShared", - [Clang<"assert_shared_capability", 0>]>]; + [Clang<"assert_shared_capability", 0>, + GNU<"assert_shared_lock">]>]; let Documentation = [AssertCapabilityDocs]; } @@ -3849,16 +3852,18 @@ def AcquireCapability : InheritableAttr { def TryAcquireCapability : InheritableAttr { let Spellings = [Clang<"try_acquire_capability", 0>, - Clang<"try_acquire_shared_capability", 0>]; - let Subjects = SubjectList<[Function], - ErrorDiag>; + Clang<"try_acquire_shared_capability", 0>, + GNU<"exclusive_trylock_function">, + GNU<"shared_trylock_function">]; + let Subjects = SubjectList<[Function]>; let LateParsed = LateAttrParseStandard; let TemplateDependent = 1; let ParseArgumentsAsUnevaluated = 1; let InheritEvenIfAlreadyPresent = 1; let Args = [ExprArgument<"SuccessValue">, VariadicExprArgument<"Args">]; let Accessors = [Accessor<"isShared", - [Clang<"try_acquire_shared_capability", 0>]>]; + [Clang<"try_acquire_shared_capability", 0>, + GNU<"shared_trylock_function">]>]; let Documentation = [TryAcquireCapabilityDocs]; } @@ -3948,54 +3953,6 @@ def AcquiredBefore : InheritableAttr { let Documentation = [Undocumented]; } -def AssertExclusiveLock : InheritableAttr { - let Spellings = [GNU<"assert_exclusive_lock">]; - let Args = [VariadicExprArgument<"Args">]; - let LateParsed = LateAttrParseStandard; - let TemplateDependent = 1; - let ParseArgumentsAsUnevaluated = 1; - let InheritEvenIfAlreadyPresent = 1; - let Subjects = SubjectList<[Function]>; - let Documentation = [Undocumented]; -} - -def AssertSharedLock : InheritableAttr { - let Spellings = [GNU<"assert_shared_lock">]; - let Args = [VariadicExprArgument<"Args">]; - let LateParsed = LateAttrParseStandard; - let TemplateDependent = 1; - let ParseArgumentsAsUnevaluated = 1; - let InheritEvenIfAlreadyPresent = 1; - let Subjects = SubjectList<[Function]>; - let Documentation = [Undocumented]; -} - -// The first argument is an integer or boolean value specifying the return value -// of a successful lock acquisition. -def ExclusiveTrylockFunction : InheritableAttr { - let Spellings = [GNU<"exclusive_trylock_function">]; - let Args = [ExprArgument<"SuccessValue">, VariadicExprArgument<"Args">]; - let LateParsed = LateAttrParseStandard; - let TemplateDependent = 1; - let ParseArgumentsAsUnevaluated = 1; - let InheritEvenIfAlreadyPresent = 1; - let Subjects = SubjectList<[Function]>; - let Documentation = [Undocumented]; -} - -// The first argument is an integer or boolean value specifying the return value -// of a successful lock acquisition. -def SharedTrylockFunction : InheritableAttr { - let Spellings = [GNU<"shared_trylock_function">]; - let Args = [ExprArgument<"SuccessValue">, VariadicExprArgument<"Args">]; - let LateParsed = LateAttrParseStandard; - let TemplateDependent = 1; - let ParseArgumentsAsUnevaluated = 1; - let InheritEvenIfAlreadyPresent = 1; - let Subjects = SubjectList<[Function]>; - let Documentation = [Undocumented]; -} - def LockReturned : InheritableAttr { let Spellings = [GNU<"lock_returned">]; let Args = [ExprArgument<"Arg">]; diff --git a/clang/lib/AST/ASTImporter.cpp b/clang/lib/AST/ASTImporter.cpp index b55b8f2c14147..00628602e61fa 100644 --- a/clang/lib/AST/ASTImporter.cpp +++ b/clang/lib/AST/ASTImporter.cpp @@ -9425,34 +9425,6 @@ Expected ASTImporter::Import(const Attr *FromAttr) { From->args_size()); break; } - case attr::AssertExclusiveLock: { - const auto *From = cast(FromAttr); - AI.importAttr(From, - AI.importArrayArg(From->args(), From->args_size()).value(), - From->args_size()); - break; - } - case attr::AssertSharedLock: { - const auto *From = cast(FromAttr); - AI.importAttr(From, - AI.importArrayArg(From->args(), From->args_size()).value(), - From->args_size()); - break; - } - case attr::ExclusiveTrylockFunction: { - const auto *From = cast(FromAttr); - AI.importAttr(From, AI.importArg(From->getSuccessValue()).value(), - AI.importArrayArg(From->args(), From->args_size()).value(), - From->args_size()); - break; - } - case attr::SharedTrylockFunction: { - const auto *From = cast(FromAttr); - AI.importAttr(From, AI.importArg(From->getSuccessValue()).value(), - AI.importArrayArg(From->args(), From->args_size()).value(), - From->args_size()); - break; - } case attr::LockReturned: { const auto *From = cast(FromAttr); AI.importAttr(From, AI.importArg(From->getArg()).value()); diff --git a/clang/lib/Analysis/ThreadSafety.cpp b/clang/lib/Analysis/ThreadSafety.cpp index 6b5b49377fa08..42fb0fe7dcdaa 100644 --- a/clang/lib/Analysis/ThreadSafety.cpp +++ b/clang/lib/Analysis/ThreadSafety.cpp @@ -1511,38 +1511,17 @@ void ThreadSafetyAnalyzer::getEdgeLockset(FactSet& Result, return; auto *FunDecl = dyn_cast_or_null(Exp->getCalleeDecl()); - if(!FunDecl || !FunDecl->hasAttrs()) + if (!FunDecl || !FunDecl->hasAttr()) return; CapExprSet ExclusiveLocksToAdd; CapExprSet SharedLocksToAdd; // If the condition is a call to a Trylock function, then grab the attributes - for (const auto *Attr : FunDecl->attrs()) { - switch (Attr->getKind()) { - case attr::TryAcquireCapability: { - auto *A = cast(Attr); - getMutexIDs(A->isShared() ? SharedLocksToAdd : ExclusiveLocksToAdd, A, - Exp, FunDecl, PredBlock, CurrBlock, A->getSuccessValue(), - Negate); - break; - }; - case attr::ExclusiveTrylockFunction: { - const auto *A = cast(Attr); - getMutexIDs(ExclusiveLocksToAdd, A, Exp, FunDecl, PredBlock, CurrBlock, - A->getSuccessValue(), Negate); - break; - } - case attr::SharedTrylockFunction: { - const auto *A = cast(Attr); - getMutexIDs(SharedLocksToAdd, A, Exp, FunDecl, PredBlock, CurrBlock, - A->getSuccessValue(), Negate); - break; - } - default: - break; - } - } + for (const auto *Attr : FunDecl->specific_attrs()) + getMutexIDs(Attr->isShared() ? SharedLocksToAdd : ExclusiveLocksToAdd, Attr, + Exp, FunDecl, PredBlock, CurrBlock, Attr->getSuccessValue(), + Negate); // Add and remove locks. SourceLocation Loc = Exp->getExprLoc(); @@ -1882,29 +1861,6 @@ void BuildLockset::handleCall(const Expr *Exp, const NamedDecl *D, // An assert will add a lock to the lockset, but will not generate // a warning if it is already there, and will not generate a warning // if it is not removed. - case attr::AssertExclusiveLock: { - const auto *A = cast(At); - - CapExprSet AssertLocks; - Analyzer->getMutexIDs(AssertLocks, A, Exp, D, Self); - for (const auto &AssertLock : AssertLocks) - Analyzer->addLock( - FSet, std::make_unique( - AssertLock, LK_Exclusive, Loc, FactEntry::Asserted)); - break; - } - case attr::AssertSharedLock: { - const auto *A = cast(At); - - CapExprSet AssertLocks; - Analyzer->getMutexIDs(AssertLocks, A, Exp, D, Self); - for (const auto &AssertLock : AssertLocks) - Analyzer->addLock( - FSet, std::make_unique( - AssertLock, LK_Shared, Loc, FactEntry::Asserted)); - break; - } - case attr::AssertCapability: { const auto *A = cast(At); CapExprSet AssertLocks; @@ -2499,12 +2455,6 @@ void ThreadSafetyAnalyzer::runAnalysis(AnalysisDeclContext &AC) { getMutexIDs(A->isShared() ? SharedLocksAcquired : ExclusiveLocksAcquired, A, nullptr, D); - } else if (isa(Attr)) { - // Don't try to check trylock functions for now. - return; - } else if (isa(Attr)) { - // Don't try to check trylock functions for now. - return; } else if (isa(Attr)) { // Don't try to check trylock functions for now. return; diff --git a/clang/lib/Sema/SemaDeclAttr.cpp b/clang/lib/Sema/SemaDeclAttr.cpp index 20ea38b7e05db..bc891fb009410 100644 --- a/clang/lib/Sema/SemaDeclAttr.cpp +++ b/clang/lib/Sema/SemaDeclAttr.cpp @@ -538,29 +538,6 @@ static bool checkLockFunAttrCommon(Sema &S, Decl *D, const ParsedAttr &AL, return true; } -static void handleAssertSharedLockAttr(Sema &S, Decl *D, const ParsedAttr &AL) { - SmallVector Args; - if (!checkLockFunAttrCommon(S, D, AL, Args)) - return; - - unsigned Size = Args.size(); - Expr **StartArg = Size == 0 ? nullptr : &Args[0]; - D->addAttr(::new (S.Context) - AssertSharedLockAttr(S.Context, AL, StartArg, Size)); -} - -static void handleAssertExclusiveLockAttr(Sema &S, Decl *D, - const ParsedAttr &AL) { - SmallVector Args; - if (!checkLockFunAttrCommon(S, D, AL, Args)) - return; - - unsigned Size = Args.size(); - Expr **StartArg = Size == 0 ? nullptr : &Args[0]; - D->addAttr(::new (S.Context) - AssertExclusiveLockAttr(S.Context, AL, StartArg, Size)); -} - /// Checks to be sure that the given parameter number is in bounds, and /// is an integral type. Will emit appropriate diagnostics if this returns /// false. @@ -640,26 +617,6 @@ static bool checkTryLockFunAttrCommon(Sema &S, Decl *D, const ParsedAttr &AL, return true; } -static void handleSharedTrylockFunctionAttr(Sema &S, Decl *D, - const ParsedAttr &AL) { - SmallVector Args; - if (!checkTryLockFunAttrCommon(S, D, AL, Args)) - return; - - D->addAttr(::new (S.Context) SharedTrylockFunctionAttr( - S.Context, AL, AL.getArgAsExpr(0), Args.data(), Args.size())); -} - -static void handleExclusiveTrylockFunctionAttr(Sema &S, Decl *D, - const ParsedAttr &AL) { - SmallVector Args; - if (!checkTryLockFunAttrCommon(S, D, AL, Args)) - return; - - D->addAttr(::new (S.Context) ExclusiveTrylockFunctionAttr( - S.Context, AL, AL.getArgAsExpr(0), Args.data(), Args.size())); -} - static void handleLockReturnedAttr(Sema &S, Decl *D, const ParsedAttr &AL) { // check that the argument is lockable object SmallVector Args; @@ -7528,12 +7485,6 @@ ProcessDeclAttribute(Sema &S, Scope *scope, Decl *D, const ParsedAttr &AL, break; // Thread safety attributes: - case ParsedAttr::AT_AssertExclusiveLock: - handleAssertExclusiveLockAttr(S, D, AL); - break; - case ParsedAttr::AT_AssertSharedLock: - handleAssertSharedLockAttr(S, D, AL); - break; case ParsedAttr::AT_PtGuardedVar: handlePtGuardedVarAttr(S, D, AL); break; @@ -7549,18 +7500,12 @@ ProcessDeclAttribute(Sema &S, Scope *scope, Decl *D, const ParsedAttr &AL, case ParsedAttr::AT_PtGuardedBy: handlePtGuardedByAttr(S, D, AL); break; - case ParsedAttr::AT_ExclusiveTrylockFunction: - handleExclusiveTrylockFunctionAttr(S, D, AL); - break; case ParsedAttr::AT_LockReturned: handleLockReturnedAttr(S, D, AL); break; case ParsedAttr::AT_LocksExcluded: handleLocksExcludedAttr(S, D, AL); break; - case ParsedAttr::AT_SharedTrylockFunction: - handleSharedTrylockFunctionAttr(S, D, AL); - break; case ParsedAttr::AT_AcquiredBefore: handleAcquiredBeforeAttr(S, D, AL); break; diff --git a/clang/lib/Sema/SemaDeclCXX.cpp b/clang/lib/Sema/SemaDeclCXX.cpp index 2247aded9384a..05991228dbfc2 100644 --- a/clang/lib/Sema/SemaDeclCXX.cpp +++ b/clang/lib/Sema/SemaDeclCXX.cpp @@ -19401,13 +19401,7 @@ bool Sema::checkThisInStaticMemberFunctionAttributes(CXXMethodDecl *Method) { Args = llvm::ArrayRef(AA->args_begin(), AA->args_size()); else if (const auto *AB = dyn_cast(A)) Args = llvm::ArrayRef(AB->args_begin(), AB->args_size()); - else if (const auto *ETLF = dyn_cast(A)) { - Arg = ETLF->getSuccessValue(); - Args = llvm::ArrayRef(ETLF->args_begin(), ETLF->args_size()); - } else if (const auto *STLF = dyn_cast(A)) { - Arg = STLF->getSuccessValue(); - Args = llvm::ArrayRef(STLF->args_begin(), STLF->args_size()); - } else if (const auto *LR = dyn_cast(A)) + else if (const auto *LR = dyn_cast(A)) Arg = LR->getArg(); else if (const auto *LE = dyn_cast(A)) Args = llvm::ArrayRef(LE->args_begin(), LE->args_size()); @@ -19415,9 +19409,10 @@ bool Sema::checkThisInStaticMemberFunctionAttributes(CXXMethodDecl *Method) { Args = llvm::ArrayRef(RC->args_begin(), RC->args_size()); else if (const auto *AC = dyn_cast(A)) Args = llvm::ArrayRef(AC->args_begin(), AC->args_size()); - else if (const auto *AC = dyn_cast(A)) + else if (const auto *AC = dyn_cast(A)) { + Arg = AC->getSuccessValue(); Args = llvm::ArrayRef(AC->args_begin(), AC->args_size()); - else if (const auto *RC = dyn_cast(A)) + } else if (const auto *RC = dyn_cast(A)) Args = llvm::ArrayRef(RC->args_begin(), RC->args_size()); if (Arg && !Finder.TraverseStmt(Arg)) diff --git a/clang/test/Sema/attr-capabilities.c b/clang/test/Sema/attr-capabilities.c index 5138803bd5eb7..12b18b687803e 100644 --- a/clang/test/Sema/attr-capabilities.c +++ b/clang/test/Sema/attr-capabilities.c @@ -14,7 +14,7 @@ struct __attribute__((capability("custom"))) CustomName {}; int Test1 __attribute__((capability("test1"))); // expected-error {{'capability' attribute only applies to structs, unions, classes, and typedefs}} int Test2 __attribute__((shared_capability("test2"))); // expected-error {{'shared_capability' attribute only applies to structs, unions, classes, and typedefs}} int Test3 __attribute__((acquire_capability("test3"))); // expected-warning {{'acquire_capability' attribute only applies to functions}} -int Test4 __attribute__((try_acquire_capability("test4"))); // expected-error {{'try_acquire_capability' attribute only applies to functions}} +int Test4 __attribute__((try_acquire_capability("test4"))); // expected-warning {{'try_acquire_capability' attribute only applies to functions}} int Test5 __attribute__((release_capability("test5"))); // expected-warning {{'release_capability' attribute only applies to functions}} struct __attribute__((capability(12))) Test3 {}; // expected-error {{expected string literal as argument of 'capability' attribute}} diff --git a/clang/unittests/AST/ASTImporterTest.cpp b/clang/unittests/AST/ASTImporterTest.cpp index 40e1197bc21f1..4192faee1af80 100644 --- a/clang/unittests/AST/ASTImporterTest.cpp +++ b/clang/unittests/AST/ASTImporterTest.cpp @@ -7977,42 +7977,6 @@ TEST_P(ImportAttributes, ImportAcquiredBefore) { checkImportVariadicArg(FromAttr->args(), ToAttr->args()); } -TEST_P(ImportAttributes, ImportAssertExclusiveLock) { - AssertExclusiveLockAttr *FromAttr, *ToAttr; - importAttr("void test(int A1, int A2) " - "__attribute__((assert_exclusive_lock(A1, A2)));", - FromAttr, ToAttr); - checkImportVariadicArg(FromAttr->args(), ToAttr->args()); -} - -TEST_P(ImportAttributes, ImportAssertSharedLock) { - AssertSharedLockAttr *FromAttr, *ToAttr; - importAttr( - "void test(int A1, int A2) __attribute__((assert_shared_lock(A1, A2)));", - FromAttr, ToAttr); - checkImportVariadicArg(FromAttr->args(), ToAttr->args()); -} - -TEST_P(ImportAttributes, ImportExclusiveTrylockFunction) { - ExclusiveTrylockFunctionAttr *FromAttr, *ToAttr; - importAttr( - "void test(int A1, int A2) __attribute__((exclusive_trylock_function(1, " - "A1, A2)));", - FromAttr, ToAttr); - checkImported(FromAttr->getSuccessValue(), ToAttr->getSuccessValue()); - checkImportVariadicArg(FromAttr->args(), ToAttr->args()); -} - -TEST_P(ImportAttributes, ImportSharedTrylockFunction) { - SharedTrylockFunctionAttr *FromAttr, *ToAttr; - importAttr( - "void test(int A1, int A2) __attribute__((shared_trylock_function(1, A1, " - "A2)));", - FromAttr, ToAttr); - checkImported(FromAttr->getSuccessValue(), ToAttr->getSuccessValue()); - checkImportVariadicArg(FromAttr->args(), ToAttr->args()); -} - TEST_P(ImportAttributes, ImportLockReturned) { LockReturnedAttr *FromAttr, *ToAttr; importAttr( From 7cb7b2d39d3b4aa984bfaeaf5e69fbfb074edd41 Mon Sep 17 00:00:00 2001 From: Alexandre Ganea Date: Tue, 15 Apr 2025 17:37:01 -0400 Subject: [PATCH 32/63] [llvm] Build Windows release package with clang-cl if possible (#135446) If `clang-cl.exe` and `lld-link.exe` are installed in `%PATH%`, the Windows release build script will now use these by default, in place of MSVC. The reason for doing this is that MSVC still has, for the past year(s), a O(N^2) behavior when building certain LLVM source files, which leads to long build times (minutes per file). A report was filled here: https://developercommunity.visualstudio.com/t/ON2-in-SparseBitVectorBase-when-com/10657991 Also added a `--force-msvc` option to the script, to use MSVC even if clang-cl is installed. --- llvm/utils/release/build_llvm_release.bat | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/llvm/utils/release/build_llvm_release.bat b/llvm/utils/release/build_llvm_release.bat index 588d7201fcb92..3042fc2d77dd1 100755 --- a/llvm/utils/release/build_llvm_release.bat +++ b/llvm/utils/release/build_llvm_release.bat @@ -7,7 +7,7 @@ goto begin echo Script for building the LLVM installer on Windows, echo used for the releases at https://github.com/llvm/llvm-project/releases echo. -echo Usage: build_llvm_release.bat --version ^ [--x86,--x64, --arm64] [--skip-checkout] [--local-python] +echo Usage: build_llvm_release.bat --version ^ [--x86,--x64, --arm64] [--skip-checkout] [--local-python] [--force-msvc] echo. echo Options: echo --version: [required] version to build @@ -17,6 +17,7 @@ echo --x64: build and test x64 variant echo --arm64: build and test arm64 variant echo --skip-checkout: use local git checkout instead of downloading src.zip echo --local-python: use installed Python and does not try to use a specific version (3.10) +echo --force-msvc: use MSVC compiler for stage0, even if clang-cl is present echo. echo Note: At least one variant to build is required. echo. @@ -34,6 +35,7 @@ set x64= set arm64= set skip-checkout= set local-python= +set force-msvc= call :parse_args %* if "%help%" NEQ "" goto usage @@ -165,6 +167,24 @@ set common_cmake_flags=^ -DLLVM_ENABLE_RPMALLOC=ON ^ -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;compiler-rt;lldb;openmp" +if "%force-msvc%" == "" ( + where /q clang-cl + if errorlevel 0 ( + where /q lld-link + if errorlevel 0 ( + set common_compiler_flags=%common_compiler_flags% -fuse-ld=lld + + set common_cmake_flags=%common_cmake_flags%^ + -DCMAKE_C_COMPILER=clang-cl.exe ^ + -DCMAKE_CXX_COMPILER=clang-cl.exe ^ + -DCMAKE_LINKER=lld-link.exe ^ + -DLLVM_ENABLE_LLD=ON ^ + -DCMAKE_C_FLAGS="%common_compiler_flags%" ^ + -DCMAKE_CXX_FLAGS="%common_compiler_flags%" + ) + ) +) + set cmake_profile_flags="" REM Preserve original path From d0372179fbbcb7b3fa680a78919a980fa4384c46 Mon Sep 17 00:00:00 2001 From: Mircea Trofin Date: Tue, 15 Apr 2025 14:43:28 -0700 Subject: [PATCH 33/63] [nfc] Add doc comment for `canReturn` in Analysis/CFG.h (#135862) --- llvm/include/llvm/Analysis/CFG.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/llvm/include/llvm/Analysis/CFG.h b/llvm/include/llvm/Analysis/CFG.h index 8451e88146d7c..052ffb2872af7 100644 --- a/llvm/include/llvm/Analysis/CFG.h +++ b/llvm/include/llvm/Analysis/CFG.h @@ -174,7 +174,10 @@ bool containsIrreducibleCFG(RPOTraversalT &RPOTraversal, const LoopInfoT &LI) { return false; } + +/// Return true if there is at least a path through which F can return, false if +/// there is no such path. bool canReturn(const Function &F); -} // End llvm namespace +} // namespace llvm #endif From 2d98bdc12c291523c3543ceaf1c526e25dcaedc6 Mon Sep 17 00:00:00 2001 From: Daniel Thornburgh Date: Tue, 15 Apr 2025 14:46:55 -0700 Subject: [PATCH 34/63] =?UTF-8?q?Revert=20"[llvm][clang]=20Allocate=20a=20?= =?UTF-8?q?new=20stack=20instead=20of=20spawning=20a=20new=20=E2=80=A6=20(?= =?UTF-8?q?#135865)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit …thread to get more stack space (#133173)" This change breaks the Clang build on Mac AArch64. This reverts commit d0c973a7a0149db3b71767d4c5a20a31e6a8ed5b. This reverts commit 429a84f8a4bf559f43f50072747ef49d3e3b2cf1. This reverts commit 4f64c80d5a23c244f942193e58ecac666c173308. --- clang/docs/ReleaseNotes.rst | 4 - clang/include/clang/Basic/Stack.h | 5 +- clang/lib/Basic/Stack.cpp | 40 ++++-- clang/lib/Frontend/CompilerInstance.cpp | 2 +- .../llvm/Support/CrashRecoveryContext.h | 3 - llvm/include/llvm/Support/ProgramStack.h | 63 ---------- llvm/include/llvm/Support/thread.h | 1 - llvm/lib/Support/CMakeLists.txt | 1 - llvm/lib/Support/CrashRecoveryContext.cpp | 11 -- llvm/lib/Support/ProgramStack.cpp | 114 ------------------ llvm/unittests/Support/CMakeLists.txt | 1 - llvm/unittests/Support/ProgramStackTest.cpp | 35 ------ 12 files changed, 30 insertions(+), 250 deletions(-) delete mode 100644 llvm/include/llvm/Support/ProgramStack.h delete mode 100644 llvm/lib/Support/ProgramStack.cpp delete mode 100644 llvm/unittests/Support/ProgramStackTest.cpp diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst index 38142ad32bea0..166f26921cb71 100644 --- a/clang/docs/ReleaseNotes.rst +++ b/clang/docs/ReleaseNotes.rst @@ -195,10 +195,6 @@ Non-comprehensive list of changes in this release - Added `__builtin_elementwise_exp10`. - For AMDPGU targets, added `__builtin_v_cvt_off_f32_i4` that maps to the `v_cvt_off_f32_i4` instruction. - Added `__builtin_elementwise_minnum` and `__builtin_elementwise_maxnum`. -- Clang itself now uses split stacks instead of threads for allocating more - stack space when running on Apple AArch64 based platforms. This means that - stack traces of Clang from debuggers, crashes, and profilers may look - different than before. New Compiler Flags ------------------ diff --git a/clang/include/clang/Basic/Stack.h b/clang/include/clang/Basic/Stack.h index 9674b9d9b62c3..30ebd94aedd1f 100644 --- a/clang/include/clang/Basic/Stack.h +++ b/clang/include/clang/Basic/Stack.h @@ -27,10 +27,7 @@ namespace clang { /// Call this once on each thread, as soon after starting the thread as /// feasible, to note the approximate address of the bottom of the stack. - /// - /// \param ForceSet set to true if you know the call is near the bottom of a - /// new stack. Used for split stacks. - void noteBottomOfStack(bool ForceSet = false); + void noteBottomOfStack(); /// Determine whether the stack is nearly exhausted. bool isStackNearlyExhausted(); diff --git a/clang/lib/Basic/Stack.cpp b/clang/lib/Basic/Stack.cpp index 8cbb84943f8d3..aa15d8e66950f 100644 --- a/clang/lib/Basic/Stack.cpp +++ b/clang/lib/Basic/Stack.cpp @@ -13,13 +13,33 @@ #include "clang/Basic/Stack.h" #include "llvm/Support/CrashRecoveryContext.h" -#include "llvm/Support/ProgramStack.h" -static LLVM_THREAD_LOCAL uintptr_t BottomOfStack = 0; +#ifdef _MSC_VER +#include // for _AddressOfReturnAddress +#endif -void clang::noteBottomOfStack(bool ForceSet) { - if (!BottomOfStack || ForceSet) - BottomOfStack = llvm::getStackPointer(); +static LLVM_THREAD_LOCAL void *BottomOfStack = nullptr; + +static void *getStackPointer() { +#if __GNUC__ || __has_builtin(__builtin_frame_address) + return __builtin_frame_address(0); +#elif defined(_MSC_VER) + return _AddressOfReturnAddress(); +#else + char CharOnStack = 0; + // The volatile store here is intended to escape the local variable, to + // prevent the compiler from optimizing CharOnStack into anything other + // than a char on the stack. + // + // Tested on: MSVC 2015 - 2019, GCC 4.9 - 9, Clang 3.2 - 9, ICC 13 - 19. + char *volatile Ptr = &CharOnStack; + return Ptr; +#endif +} + +void clang::noteBottomOfStack() { + if (!BottomOfStack) + BottomOfStack = getStackPointer(); } bool clang::isStackNearlyExhausted() { @@ -31,8 +51,7 @@ bool clang::isStackNearlyExhausted() { if (!BottomOfStack) return false; - intptr_t StackDiff = - (intptr_t)llvm::getStackPointer() - (intptr_t)BottomOfStack; + intptr_t StackDiff = (intptr_t)getStackPointer() - (intptr_t)BottomOfStack; size_t StackUsage = (size_t)std::abs(StackDiff); // If the stack pointer has a surprising value, we do not understand this @@ -47,12 +66,9 @@ bool clang::isStackNearlyExhausted() { void clang::runWithSufficientStackSpaceSlow(llvm::function_ref Diag, llvm::function_ref Fn) { llvm::CrashRecoveryContext CRC; - // Preserve the BottomOfStack in case RunSafelyOnNewStack uses split stacks. - uintptr_t PrevBottom = BottomOfStack; - CRC.RunSafelyOnNewStack([&] { - noteBottomOfStack(true); + CRC.RunSafelyOnThread([&] { + noteBottomOfStack(); Diag(); Fn(); }, DesiredStackSize); - BottomOfStack = PrevBottom; } diff --git a/clang/lib/Frontend/CompilerInstance.cpp b/clang/lib/Frontend/CompilerInstance.cpp index 5fe80fc16482e..243e0a3c15b05 100644 --- a/clang/lib/Frontend/CompilerInstance.cpp +++ b/clang/lib/Frontend/CompilerInstance.cpp @@ -1265,7 +1265,7 @@ bool CompilerInstance::compileModule(SourceLocation ImportLoc, // Execute the action to actually build the module in-place. Use a separate // thread so that we get a stack large enough. - bool Crashed = !llvm::CrashRecoveryContext().RunSafelyOnNewStack( + bool Crashed = !llvm::CrashRecoveryContext().RunSafelyOnThread( [&]() { GenerateModuleFromModuleMapAction Action; Instance.ExecuteAction(Action); diff --git a/llvm/include/llvm/Support/CrashRecoveryContext.h b/llvm/include/llvm/Support/CrashRecoveryContext.h index 31293d6715757..26ddf97b3ef02 100644 --- a/llvm/include/llvm/Support/CrashRecoveryContext.h +++ b/llvm/include/llvm/Support/CrashRecoveryContext.h @@ -97,9 +97,6 @@ class CrashRecoveryContext { return RunSafelyOnThread([&]() { Fn(UserData); }, RequestedStackSize); } - bool RunSafelyOnNewStack(function_ref, - unsigned RequestedStackSize = 0); - /// Explicitly trigger a crash recovery in the current process, and /// return failure from RunSafely(). This function does not return. [[noreturn]] void HandleExit(int RetCode); diff --git a/llvm/include/llvm/Support/ProgramStack.h b/llvm/include/llvm/Support/ProgramStack.h deleted file mode 100644 index 232a7b5670b44..0000000000000 --- a/llvm/include/llvm/Support/ProgramStack.h +++ /dev/null @@ -1,63 +0,0 @@ -//===--- ProgramStack.h -----------------------------------------*- C++ -*-===// -// -// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. -// See https://llvm.org/LICENSE.txt for license information. -// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception -// -//===----------------------------------------------------------------------===// - -#ifndef LLVM_SUPPORT_PROGRAMSTACK_H -#define LLVM_SUPPORT_PROGRAMSTACK_H - -#include "llvm/ADT/STLFunctionalExtras.h" - -// LLVM_HAS_SPLIT_STACKS is exposed in the header because CrashRecoveryContext -// needs to know if it's running on another thread or not. -// -// Currently only Apple AArch64 is known to support split stacks in the debugger -// and other tooling. -#if defined(__APPLE__) && defined(__aarch64__) && \ - LLVM_HAS_CPP_ATTRIBUTE(gnu::naked) && __has_extension(gnu_asm) -# define LLVM_HAS_SPLIT_STACKS -# define LLVM_HAS_SPLIT_STACKS_AARCH64 -#endif - -namespace llvm { - -/// \returns an address close to the current value of the stack pointer. -/// -/// The value is not guaranteed to point to anything specific. It can be used to -/// estimate how much stack space has been used since the previous call. -uintptr_t getStackPointer(); - -/// \returns the default stack size for this platform. -/// -/// Based on \p RLIMIT_STACK or the equivalent. -unsigned getDefaultStackSize(); - -/// Runs Fn on a new stack of at least the given size. -/// -/// \param StackSize requested stack size. A size of 0 uses the default stack -/// size of the platform. -/// -/// The preferred implementation is split stacks on platforms that have a good -/// debugging experience for them. On other platforms a new thread is used. -void runOnNewStack(unsigned StackSize, function_ref Fn); - -template -std::enable_if_t, R> -runOnNewStack(unsigned StackSize, function_ref Fn, Ts &&...Args) { - std::optional Ret; - runOnNewStack(StackSize, [&]() { Ret = Fn(std::forward(Args)...); }); - return std::move(*Ret); -} - -template -void runOnNewStack(unsigned StackSize, function_ref Fn, - Ts &&...Args) { - runOnNewStack(StackSize, [&]() { Fn(std::forward(Args)...); }); -} - -} // namespace llvm - -#endif // LLVM_SUPPORT_PROGRAMSTACK_H diff --git a/llvm/include/llvm/Support/thread.h b/llvm/include/llvm/Support/thread.h index ef2fba822cb1c..e3005fdb63175 100644 --- a/llvm/include/llvm/Support/thread.h +++ b/llvm/include/llvm/Support/thread.h @@ -213,7 +213,6 @@ inline thread::id get_id() { return std::this_thread::get_id(); } #else // !LLVM_ENABLE_THREADS -#include "llvm/Support/ErrorHandling.h" #include namespace llvm { diff --git a/llvm/lib/Support/CMakeLists.txt b/llvm/lib/Support/CMakeLists.txt index def37f3f278d0..98ffd829d80b8 100644 --- a/llvm/lib/Support/CMakeLists.txt +++ b/llvm/lib/Support/CMakeLists.txt @@ -295,7 +295,6 @@ add_llvm_component_library(LLVMSupport Path.cpp Process.cpp Program.cpp - ProgramStack.cpp RWMutex.cpp Signals.cpp Threading.cpp diff --git a/llvm/lib/Support/CrashRecoveryContext.cpp b/llvm/lib/Support/CrashRecoveryContext.cpp index 88c38d7526e71..f53aea177d612 100644 --- a/llvm/lib/Support/CrashRecoveryContext.cpp +++ b/llvm/lib/Support/CrashRecoveryContext.cpp @@ -10,7 +10,6 @@ #include "llvm/Config/llvm-config.h" #include "llvm/Support/ErrorHandling.h" #include "llvm/Support/ExitCodes.h" -#include "llvm/Support/ProgramStack.h" #include "llvm/Support/Signals.h" #include "llvm/Support/thread.h" #include @@ -524,13 +523,3 @@ bool CrashRecoveryContext::RunSafelyOnThread(function_ref Fn, CRC->setSwitchedThread(); return Info.Result; } - -bool CrashRecoveryContext::RunSafelyOnNewStack(function_ref Fn, - unsigned RequestedStackSize) { -#ifdef LLVM_HAS_SPLIT_STACKS - return runOnNewStack(RequestedStackSize, - function_ref([&]() { return RunSafely(Fn); })); -#else - return RunSafelyOnThread(Fn, RequestedStackSize); -#endif -} diff --git a/llvm/lib/Support/ProgramStack.cpp b/llvm/lib/Support/ProgramStack.cpp deleted file mode 100644 index 9e5a546b34974..0000000000000 --- a/llvm/lib/Support/ProgramStack.cpp +++ /dev/null @@ -1,114 +0,0 @@ -//===--- RunOnNewStack.cpp - Crash Recovery -------------------------------===// -// -// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. -// See https://llvm.org/LICENSE.txt for license information. -// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception -// -//===----------------------------------------------------------------------===// - -#include "llvm/Support/ProgramStack.h" -#include "llvm/Config/config.h" -#include "llvm/Support/Compiler.h" - -#ifdef LLVM_ON_UNIX -# include // for getrlimit -#endif - -#ifdef _MSC_VER -# include // for _AddressOfReturnAddress -#endif - -#ifndef LLVM_HAS_SPLIT_STACKS -# include "llvm/Support/thread.h" -#endif - -using namespace llvm; - -uintptr_t llvm::getStackPointer() { -#if __GNUC__ || __has_builtin(__builtin_frame_address) - return (uintptr_t)__builtin_frame_address(0); -#elif defined(_MSC_VER) - return (uintptr_t)_AddressOfReturnAddress(); -#else - volatile char CharOnStack = 0; - // The volatile store here is intended to escape the local variable, to - // prevent the compiler from optimizing CharOnStack into anything other - // than a char on the stack. - // - // Tested on: MSVC 2015 - 2019, GCC 4.9 - 9, Clang 3.2 - 9, ICC 13 - 19. - char *volatile Ptr = &CharOnStack; - return (uintptr_t)Ptr; -#endif -} - -unsigned llvm::getDefaultStackSize() { -#ifdef LLVM_ON_UNIX - rlimit RL; - getrlimit(RLIMIT_STACK, &RL); - return RL.rlim_cur; -#else - // Clang recursively parses, instantiates templates, and evaluates constant - // expressions. We've found 8MiB to be a reasonable stack size given the way - // Clang works and the way C++ is commonly written. - return 8 << 20; -#endif -} - -namespace { -#ifdef LLVM_HAS_SPLIT_STACKS_AARCH64 -[[gnu::naked]] void runOnNewStackImpl(void *Stack, void (*Fn)(void *), - void *Ctx) { - __asm__ volatile( - "mov x16, sp\n\t" - "sub x0, x0, #0x20\n\t" // subtract space from stack - "stp xzr, x16, [x0, #0x00]\n\t" // save old sp - "stp x29, x30, [x0, #0x10]\n\t" // save fp, lr - "mov sp, x0\n\t" // switch to new stack - "add x29, x0, #0x10\n\t" // switch to new frame - ".cfi_def_cfa w29, 16\n\t" - ".cfi_offset w30, -8\n\t" // lr - ".cfi_offset w29, -16\n\t" // fp - - "mov x0, x2\n\t" // Ctx is the only argument - "blr x1\n\t" // call Fn - - "ldp x29, x30, [sp, #0x10]\n\t" // restore fp, lr - "ldp xzr, x16, [sp, #0x00]\n\t" // load old sp - "mov sp, x16\n\t" - "ret" - ); -} -#endif - -#ifdef LLVM_HAS_SPLIT_STACKS -void callback(void *Ctx) { - (*reinterpret_cast *>(Ctx))(); -} -#endif -} // namespace - -#ifdef LLVM_HAS_SPLIT_STACKS -void llvm::runOnNewStack(unsigned StackSize, function_ref Fn) { - if (StackSize == 0) - StackSize = getDefaultStackSize(); - - // We use malloc here instead of mmap because: - // - it's simpler, - // - many malloc implementations will reuse the allocation in cases where - // we're bouncing accross the edge of a stack boundry, and - // - many malloc implemenations will already provide guard pages for - // allocations this large. - void *Stack = malloc(StackSize); - void *BottomOfStack = (char *)Stack + StackSize; - - runOnNewStackImpl(BottomOfStack, callback, &Fn); - - free(Stack); -} -#else -void llvm::runOnNewStack(unsigned StackSize, function_ref Fn) { - llvm::thread Thread( - StackSize == 0 ? std::nullopt : std::optional(StackSize), Fn); - Thread.join(); -} -#endif diff --git a/llvm/unittests/Support/CMakeLists.txt b/llvm/unittests/Support/CMakeLists.txt index e5bf820fb4d1c..6c4e7cb689b20 100644 --- a/llvm/unittests/Support/CMakeLists.txt +++ b/llvm/unittests/Support/CMakeLists.txt @@ -70,7 +70,6 @@ add_llvm_unittest(SupportTests PerThreadBumpPtrAllocatorTest.cpp ProcessTest.cpp ProgramTest.cpp - ProgramStackTest.cpp RecyclerTest.cpp RegexTest.cpp ReverseIterationTest.cpp diff --git a/llvm/unittests/Support/ProgramStackTest.cpp b/llvm/unittests/Support/ProgramStackTest.cpp deleted file mode 100644 index 31dfb3b88ade6..0000000000000 --- a/llvm/unittests/Support/ProgramStackTest.cpp +++ /dev/null @@ -1,35 +0,0 @@ -//===- unittest/Support/ProgramStackTest.cpp ------------------------------===// -// -// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. -// See https://llvm.org/LICENSE.txt for license information. -// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception -// -//===----------------------------------------------------------------------===// - -#include "llvm/Support/ProgramStack.h" -#include "llvm/Support/Process.h" -#include "gtest/gtest.h" - -using namespace llvm; - -static uintptr_t func(int &A) { - A = 7; - return getStackPointer(); -} - -static void func2(int &A) { - A = 5; -} - -TEST(ProgramStackTest, runOnNewStack) { - int A = 0; - uintptr_t Stack = runOnNewStack(0, function_ref(func), A); - EXPECT_EQ(A, 7); - intptr_t StackDiff = (intptr_t)llvm::getStackPointer() - (intptr_t)Stack; - size_t StackDistance = (size_t)std::abs(StackDiff); - // Page size is used as it's large enough to guarantee were not on the same - // stack but not too large to cause spurious failures. - EXPECT_GT(StackDistance, llvm::sys::Process::getPageSizeEstimate()); - runOnNewStack(0, function_ref(func2), A); - EXPECT_EQ(A, 5); -} From 8ed397d8e4d014ecc5df89a9d908c5808f201b65 Mon Sep 17 00:00:00 2001 From: Craig Topper Date: Tue, 15 Apr 2025 14:58:15 -0700 Subject: [PATCH 35/63] [DAGCombiner] Disable narrowExtractedVectorLoad for indexed loads. (#135847) The later code does not expect or preserve the index output. Fixes #135821 --- llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp | 2 +- llvm/test/CodeGen/AArch64/pr135821.ll | 27 +++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) create mode 100644 llvm/test/CodeGen/AArch64/pr135821.ll diff --git a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp index b322fe670d4a7..d72be359867ca 100644 --- a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp +++ b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp @@ -25183,7 +25183,7 @@ static SDValue narrowExtractedVectorLoad(SDNode *Extract, SelectionDAG &DAG) { return SDValue(); auto *Ld = dyn_cast(Extract->getOperand(0)); - if (!Ld || Ld->getExtensionType() || !Ld->isSimple()) + if (!Ld || !ISD::isNormalLoad(Ld) || !Ld->isSimple()) return SDValue(); // Allow targets to opt-out. diff --git a/llvm/test/CodeGen/AArch64/pr135821.ll b/llvm/test/CodeGen/AArch64/pr135821.ll new file mode 100644 index 0000000000000..cfd6cd086e130 --- /dev/null +++ b/llvm/test/CodeGen/AArch64/pr135821.ll @@ -0,0 +1,27 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 +; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu | FileCheck %s + +define <4 x float> @f(ptr %0) { +; CHECK-LABEL: f: +; CHECK: // %bb.0: +; CHECK-NEXT: sub sp, sp, #32 +; CHECK-NEXT: str x30, [sp, #16] // 8-byte Folded Spill +; CHECK-NEXT: .cfi_def_cfa_offset 32 +; CHECK-NEXT: .cfi_offset w30, -16 +; CHECK-NEXT: ldr q1, [x0, #56]! +; CHECK-NEXT: ldr d0, [x0, #16] +; CHECK-NEXT: mov v1.d[1], v0.d[0] +; CHECK-NEXT: str q1, [sp] // 16-byte Folded Spill +; CHECK-NEXT: bl use +; CHECK-NEXT: ldr q0, [sp] // 16-byte Folded Reload +; CHECK-NEXT: ldr x30, [sp, #16] // 8-byte Folded Reload +; CHECK-NEXT: add sp, sp, #32 +; CHECK-NEXT: ret + %2 = getelementptr inbounds nuw i8, ptr %0, i64 56 + %3 = load <6 x float>, ptr %2, align 4 + %4 = shufflevector <6 x float> %3, <6 x float> poison, <4 x i32> + tail call void @use(ptr %2) + ret <4 x float> %4 +} + +declare void @use(ptr) From a6208ce4c15142c26c6b73651bf466ae6b470cb0 Mon Sep 17 00:00:00 2001 From: Mircea Trofin Date: Tue, 15 Apr 2025 15:07:03 -0700 Subject: [PATCH 36/63] [nfc] move `isPresplitCoroSuspendExitEdge` to Analysis/CFG (#135849) --- llvm/include/llvm/Analysis/CFG.h | 19 +++++++++++++++++++ .../llvm/Transforms/Utils/BasicBlockUtils.h | 18 ------------------ llvm/lib/Analysis/CFG.cpp | 13 +++++++++++++ .../Instrumentation/InstrProfiling.cpp | 1 + .../Instrumentation/PGOCtxProfFlattening.cpp | 1 + llvm/lib/Transforms/Utils/BasicBlockUtils.cpp | 12 ------------ 6 files changed, 34 insertions(+), 30 deletions(-) diff --git a/llvm/include/llvm/Analysis/CFG.h b/llvm/include/llvm/Analysis/CFG.h index 052ffb2872af7..64e2079df9db2 100644 --- a/llvm/include/llvm/Analysis/CFG.h +++ b/llvm/include/llvm/Analysis/CFG.h @@ -175,6 +175,25 @@ bool containsIrreducibleCFG(RPOTraversalT &RPOTraversal, const LoopInfoT &LI) { return false; } +// Returns true if these basic blocks belong to a presplit coroutine and the +// edge corresponds to the 'default' case in the switch statement in the +// pattern: +// +// %0 = call i8 @llvm.coro.suspend(token none, i1 false) +// switch i8 %0, label %suspend [i8 0, label %resume +// i8 1, label %cleanup] +// +// i.e. the edge to the `%suspend` BB. This edge is special in that it will +// be elided by coroutine lowering (coro-split), and the `%suspend` BB needs +// to be kept as-is. It's not a real CFG edge - post-lowering, it will end +// up being a `ret`, and it must be thus lowerable to support symmetric +// transfer. For example: +// - this edge is not a loop exit edge if encountered in a loop (and should +// be ignored) +// - must not be split for PGO instrumentation, for example. +bool isPresplitCoroSuspendExitEdge(const BasicBlock &Src, + const BasicBlock &Dest); + /// Return true if there is at least a path through which F can return, false if /// there is no such path. bool canReturn(const Function &F); diff --git a/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h b/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h index 6faff3d1fd8e3..adc1851c2ec2f 100644 --- a/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h +++ b/llvm/include/llvm/Transforms/Utils/BasicBlockUtils.h @@ -610,24 +610,6 @@ void InvertBranch(BranchInst *PBI, IRBuilderBase &Builder); // br/brcond/unreachable/ret bool hasOnlySimpleTerminator(const Function &F); -// Returns true if these basic blocks belong to a presplit coroutine and the -// edge corresponds to the 'default' case in the switch statement in the -// pattern: -// -// %0 = call i8 @llvm.coro.suspend(token none, i1 false) -// switch i8 %0, label %suspend [i8 0, label %resume -// i8 1, label %cleanup] -// -// i.e. the edge to the `%suspend` BB. This edge is special in that it will -// be elided by coroutine lowering (coro-split), and the `%suspend` BB needs -// to be kept as-is. It's not a real CFG edge - post-lowering, it will end -// up being a `ret`, and it must be thus lowerable to support symmetric -// transfer. For example: -// - this edge is not a loop exit edge if encountered in a loop (and should -// be ignored) -// - must not be split for PGO instrumentation, for example. -bool isPresplitCoroSuspendExitEdge(const BasicBlock &Src, - const BasicBlock &Dest); } // end namespace llvm #endif // LLVM_TRANSFORMS_UTILS_BASICBLOCKUTILS_H diff --git a/llvm/lib/Analysis/CFG.cpp b/llvm/lib/Analysis/CFG.cpp index 8ced4a901557d..0d32e101ee0b4 100644 --- a/llvm/lib/Analysis/CFG.cpp +++ b/llvm/lib/Analysis/CFG.cpp @@ -14,6 +14,7 @@ #include "llvm/Analysis/CFG.h" #include "llvm/Analysis/LoopInfo.h" #include "llvm/IR/Dominators.h" +#include "llvm/IR/IntrinsicInst.h" #include "llvm/Support/CommandLine.h" using namespace llvm; @@ -356,3 +357,15 @@ bool llvm::canReturn(const Function &F) { return false; } + +bool llvm::isPresplitCoroSuspendExitEdge(const BasicBlock &Src, + const BasicBlock &Dest) { + assert(Src.getParent() == Dest.getParent()); + if (!Src.getParent()->isPresplitCoroutine()) + return false; + if (auto *SW = dyn_cast(Src.getTerminator())) + if (auto *Intr = dyn_cast(SW->getCondition())) + return Intr->getIntrinsicID() == Intrinsic::coro_suspend && + SW->getDefaultDest() == &Dest; + return false; +} \ No newline at end of file diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp index 008c1faf0a0c3..84bf4c62c7aad 100644 --- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp +++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp @@ -20,6 +20,7 @@ #include "llvm/ADT/Twine.h" #include "llvm/Analysis/BlockFrequencyInfo.h" #include "llvm/Analysis/BranchProbabilityInfo.h" +#include "llvm/Analysis/CFG.h" #include "llvm/Analysis/LoopInfo.h" #include "llvm/Analysis/TargetLibraryInfo.h" #include "llvm/IR/Attributes.h" diff --git a/llvm/lib/Transforms/Instrumentation/PGOCtxProfFlattening.cpp b/llvm/lib/Transforms/Instrumentation/PGOCtxProfFlattening.cpp index 05f364a974c6c..508a41684ed20 100644 --- a/llvm/lib/Transforms/Instrumentation/PGOCtxProfFlattening.cpp +++ b/llvm/lib/Transforms/Instrumentation/PGOCtxProfFlattening.cpp @@ -21,6 +21,7 @@ #include "llvm/Transforms/Instrumentation/PGOCtxProfFlattening.h" #include "llvm/ADT/STLExtras.h" #include "llvm/ADT/ScopeExit.h" +#include "llvm/Analysis/CFG.h" #include "llvm/Analysis/CtxProfAnalysis.h" #include "llvm/Analysis/ProfileSummaryInfo.h" #include "llvm/IR/Analysis.h" diff --git a/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp b/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp index 6f36e24000aa5..b78270f6309ff 100644 --- a/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp +++ b/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp @@ -1916,15 +1916,3 @@ bool llvm::hasOnlySimpleTerminator(const Function &F) { } return true; } - -bool llvm::isPresplitCoroSuspendExitEdge(const BasicBlock &Src, - const BasicBlock &Dest) { - assert(Src.getParent() == Dest.getParent()); - if (!Src.getParent()->isPresplitCoroutine()) - return false; - if (auto *SW = dyn_cast(Src.getTerminator())) - if (auto *Intr = dyn_cast(SW->getCondition())) - return Intr->getIntrinsicID() == Intrinsic::coro_suspend && - SW->getDefaultDest() == &Dest; - return false; -} From 6e2bca840df9dfcffc5068c1ad0c9575f0c57e76 Mon Sep 17 00:00:00 2001 From: LLVM GN Syncbot Date: Tue, 15 Apr 2025 22:07:57 +0000 Subject: [PATCH 37/63] [gn build] Port 2d98bdc12c29 --- llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn | 1 - llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn | 1 - 2 files changed, 2 deletions(-) diff --git a/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn b/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn index 0d2330cba6a7a..3a9f43b1070a7 100644 --- a/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn +++ b/llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn @@ -124,7 +124,6 @@ static_library("Support") { "Parallel.cpp", "PluginLoader.cpp", "PrettyStackTrace.cpp", - "ProgramStack.cpp", "RISCVAttributeParser.cpp", "RISCVAttributes.cpp", "RISCVISAUtils.cpp", diff --git a/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn b/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn index 19418ad52147b..bf6a0b7523279 100644 --- a/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn +++ b/llvm/utils/gn/secondary/llvm/unittests/Support/BUILD.gn @@ -71,7 +71,6 @@ unittest("SupportTests") { "Path.cpp", "PerThreadBumpPtrAllocatorTest.cpp", "ProcessTest.cpp", - "ProgramStackTest.cpp", "ProgramTest.cpp", "RISCVAttributeParserTest.cpp", "RecyclerTest.cpp", From 31f39c83259401a26b3660dd75f645002258571d Mon Sep 17 00:00:00 2001 From: Jun Wang Date: Tue, 15 Apr 2025 15:17:33 -0700 Subject: [PATCH 38/63] [AMDGPU] Remove the AnnotateKernelFeatures pass (#130198) Previously the AnnotateKernelFeatures pass infers two attributes: amdgpu-calls and amdgpu-stack-objects, which are used to help determine if flat scratch init is allowed. PR #118907 created the amdgpu-no-flat-scratch-init attribute. Continuing with that work, this patch makes use of this attribute to determine flat scratch init, replacing amdgpu-calls and amdgpu-stack-objects. This also leads to the removal of the AnnotateKernelFeatures pass. --- llvm/lib/Target/AMDGPU/AMDGPU.h | 3 - .../AMDGPU/AMDGPUAnnotateKernelFeatures.cpp | 9 - llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def | 6 - .../lib/Target/AMDGPU/AMDGPUTargetMachine.cpp | 7 - llvm/lib/Target/AMDGPU/GCNSubtarget.cpp | 15 +- .../AMDGPU/GlobalISel/atomicrmw_udec_wrap.ll | 368 ++++- .../AMDGPU/GlobalISel/atomicrmw_uinc_wrap.ll | 390 ++++- .../AMDGPU/GlobalISel/extractelement.ll | 71 +- .../AMDGPU/GlobalISel/flat-scratch-init.ll | 4 +- ...licit-kernarg-backend-usage-global-isel.ll | 30 +- .../GlobalISel/insertelement-stack-lower.ll | 2 +- .../AMDGPU/GlobalISel/lds-global-value.ll | 5 +- .../GlobalISel/llvm.amdgcn.if.break.i64.ll | 3 + .../GlobalISel/llvm.amdgcn.trig.preop.ll | 24 + .../test/CodeGen/AMDGPU/GlobalISel/sdivrem.ll | 33 + .../test/CodeGen/AMDGPU/GlobalISel/udivrem.ll | 33 + .../abi-attribute-hints-undefined-behavior.ll | 18 +- .../AMDGPU/addrspacecast-constantexpr.ll | 62 - llvm/test/CodeGen/AMDGPU/always-uniform.ll | 3 + ...amdgpu-codegenprepare-fold-binop-select.ll | 3 + .../CodeGen/AMDGPU/amdhsa-trap-num-sgprs.ll | 4 +- .../annotate-kernel-features-hsa-call.ll | 331 ---- .../AMDGPU/annotate-kernel-features-hsa.ll | 165 -- .../AMDGPU/annotate-kernel-features.ll | 103 -- .../attr-amdgpu-flat-work-group-size.ll | 4 +- .../CodeGen/AMDGPU/attr-amdgpu-num-sgpr.ll | 4 +- .../AMDGPU/attr-amdgpu-waves-per-eu.ll | 4 +- .../attributor-flatscratchinit-globalisel.ll | 21 +- ...utor-flatscratchinit-undefined-behavior.ll | 63 + ...tor-flatscratchinit-undefined-behavior2.ll | 870 +++++++++++ llvm/test/CodeGen/AMDGPU/attributor-noopt.ll | 4 +- .../callee-special-input-sgprs-fixed-abi.ll | 40 +- llvm/test/CodeGen/AMDGPU/code-object-v3.ll | 8 +- .../CodeGen/AMDGPU/combine-reg-or-const.ll | 3 + ...dagcomb-extract-vec-elt-different-sizes.ll | 2 + .../AMDGPU/duplicate-attribute-indirect.ll | 13 - ...cannot-create-empty-or-backward-segment.ll | 2 +- .../expand-scalar-carry-out-select-user.ll | 3 + .../CodeGen/AMDGPU/extract_vector_elt-i8.ll | 100 +- llvm/test/CodeGen/AMDGPU/fabs.f16.ll | 66 + .../fast-unaligned-load-store.global.ll | 20 +- llvm/test/CodeGen/AMDGPU/fcanonicalize.ll | 236 ++- .../flat-for-global-subtarget-feature.ll | 8 +- llvm/test/CodeGen/AMDGPU/flat-scratch-reg.ll | 80 +- .../AMDGPU/fmul-2-combine-multi-use.ll | 48 + llvm/test/CodeGen/AMDGPU/fneg-fabs.f16.ll | 60 + .../CodeGen/AMDGPU/fneg-modifier-casting.ll | 3 + llvm/test/CodeGen/AMDGPU/fneg.f16.ll | 62 +- llvm/test/CodeGen/AMDGPU/half.ll | 231 +++ .../AMDGPU/hsa-metadata-kernel-code-props.ll | 4 +- llvm/test/CodeGen/AMDGPU/hsa.ll | 4 +- .../AMDGPU/implicit-kernarg-backend-usage.ll | 10 +- llvm/test/CodeGen/AMDGPU/inline-asm.i128.ll | 37 +- .../AMDGPU/insert_vector_elt.v2bf16.ll | 58 +- .../CodeGen/AMDGPU/insert_vector_elt.v2i16.ll | 214 ++- .../CodeGen/AMDGPU/invalid-addrspacecast.ll | 3 + .../CodeGen/AMDGPU/invalid-cast-load-i1.ll | 2 + llvm/test/CodeGen/AMDGPU/kernarg-size.ll | 2 +- llvm/test/CodeGen/AMDGPU/llc-pipeline.ll | 30 +- .../CodeGen/AMDGPU/llvm.amdgcn.is.private.ll | 12 + .../CodeGen/AMDGPU/llvm.amdgcn.is.shared.ll | 12 + .../AMDGPU/llvm.amdgcn.lds.kernel.id.ll | 8 +- .../AMDGPU/llvm.amdgcn.readfirstlane.ll | 70 +- .../CodeGen/AMDGPU/llvm.amdgcn.readlane.ll | 114 +- .../CodeGen/AMDGPU/llvm.amdgcn.writelane.ll | 126 +- llvm/test/CodeGen/AMDGPU/load-constant-f64.ll | 6 + llvm/test/CodeGen/AMDGPU/load-constant-i16.ll | 125 +- llvm/test/CodeGen/AMDGPU/load-constant-i32.ll | 83 +- llvm/test/CodeGen/AMDGPU/load-constant-i64.ll | 18 + llvm/test/CodeGen/AMDGPU/load-constant-i8.ll | 164 +- llvm/test/CodeGen/AMDGPU/load-global-i16.ll | 129 +- llvm/test/CodeGen/AMDGPU/load-global-i32.ll | 105 +- llvm/test/CodeGen/AMDGPU/load-select-ptr.ll | 3 +- .../CodeGen/AMDGPU/mad24-get-global-id.ll | 2 +- .../match-perm-extract-vector-elt-bug.ll | 8 +- llvm/test/CodeGen/AMDGPU/memcpy-libcall.ll | 300 ++-- .../AMDGPU/memory-legalizer-flat-agent.ll | 1380 +++++++++++++++++ .../memory-legalizer-flat-nontemporal.ll | 75 + .../memory-legalizer-flat-singlethread.ll | 1380 +++++++++++++++++ .../AMDGPU/memory-legalizer-flat-system.ll | 1380 +++++++++++++++++ .../AMDGPU/memory-legalizer-flat-volatile.ll | 66 + .../AMDGPU/memory-legalizer-flat-wavefront.ll | 1365 ++++++++++++++++ .../AMDGPU/memory-legalizer-flat-workgroup.ll | 1320 ++++++++++++++++ .../AMDGPU/memory-legalizer-global-agent.ll | 273 ++++ .../memory-legalizer-global-nontemporal.ll | 15 + .../memory-legalizer-global-singlethread.ll | 276 ++++ .../AMDGPU/memory-legalizer-global-system.ll | 261 ++++ .../memory-legalizer-global-volatile.ll | 18 + .../memory-legalizer-global-wavefront.ll | 276 ++++ .../memory-legalizer-global-workgroup.ll | 276 ++++ .../memory-legalizer-local-nontemporal.ll | 9 + .../AMDGPU/memory-legalizer-local-volatile.ll | 6 + .../memory-legalizer-private-nontemporal.ll | 59 +- .../memory-legalizer-private-volatile.ll | 30 +- llvm/test/CodeGen/AMDGPU/min.ll | 210 +++ llvm/test/CodeGen/AMDGPU/pack.v2f16.ll | 21 + llvm/test/CodeGen/AMDGPU/pack.v2i16.ll | 18 + .../AMDGPU/pal-simple-indirect-call.ll | 8 - ...al-regcopy-and-spill-missed-at-regalloc.ll | 51 +- .../AMDGPU/preload-implicit-kernargs.ll | 178 +-- llvm/test/CodeGen/AMDGPU/preload-kernargs.ll | 379 +++-- llvm/test/CodeGen/AMDGPU/sad.ll | 114 +- .../CodeGen/AMDGPU/scalar_to_vector.v8i16.ll | 16 + .../scc-clobbered-sgpr-to-vmem-spill.ll | 464 +++--- .../CodeGen/AMDGPU/sgpr-spill-no-vgprs.ll | 2 +- llvm/test/CodeGen/AMDGPU/shift-i128.ll | 24 +- .../CodeGen/AMDGPU/simple-indirect-call.ll | 15 - llvm/test/CodeGen/AMDGPU/sint_to_fp.f64.ll | 70 +- .../CodeGen/AMDGPU/spill-vector-superclass.ll | 6 +- llvm/test/CodeGen/AMDGPU/store-weird-sizes.ll | 6 + llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll | 2 +- llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll | 2 +- llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll | 2 +- llvm/test/CodeGen/AMDGPU/trap-abis.ll | 16 +- llvm/test/CodeGen/AMDGPU/udiv.ll | 45 + llvm/test/CodeGen/AMDGPU/uint_to_fp.f64.ll | 91 +- .../AMDGPU/vgpr-spill-placement-issue61083.ll | 2 +- ...ine-function-info-long-branch-reg-debug.ll | 2 +- .../machine-function-info-long-branch-reg.ll | 2 +- 119 files changed, 13654 insertions(+), 1853 deletions(-) create mode 100644 llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior.ll create mode 100644 llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior2.ll diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.h b/llvm/lib/Target/AMDGPU/AMDGPU.h index 03cd45d7de6f2..4ff761ec19b3c 100644 --- a/llvm/lib/Target/AMDGPU/AMDGPU.h +++ b/llvm/lib/Target/AMDGPU/AMDGPU.h @@ -95,11 +95,8 @@ void initializeAMDGPUDAGToDAGISelLegacyPass(PassRegistry &); void initializeAMDGPUAlwaysInlinePass(PassRegistry&); -Pass *createAMDGPUAnnotateKernelFeaturesPass(); Pass *createAMDGPUAttributorLegacyPass(); void initializeAMDGPUAttributorLegacyPass(PassRegistry &); -void initializeAMDGPUAnnotateKernelFeaturesPass(PassRegistry &); -extern char &AMDGPUAnnotateKernelFeaturesID; // DPP/Iterative option enables the atomic optimizer with given strategy // whereas None disables the atomic optimizer. diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateKernelFeatures.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateKernelFeatures.cpp index a9bd41382c255..9c9fa5c6e2f0f 100644 --- a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateKernelFeatures.cpp +++ b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateKernelFeatures.cpp @@ -52,11 +52,6 @@ class AMDGPUAnnotateKernelFeatures : public CallGraphSCCPass { char AMDGPUAnnotateKernelFeatures::ID = 0; -char &llvm::AMDGPUAnnotateKernelFeaturesID = AMDGPUAnnotateKernelFeatures::ID; - -INITIALIZE_PASS(AMDGPUAnnotateKernelFeatures, DEBUG_TYPE, - "Add AMDGPU function attributes", false, false) - bool AMDGPUAnnotateKernelFeatures::addFeatureAttributes(Function &F) { bool HaveStackObjects = false; bool Changed = false; @@ -131,7 +126,3 @@ bool AMDGPUAnnotateKernelFeatures::doInitialization(CallGraph &CG) { TM = &TPC->getTM(); return false; } - -Pass *llvm::createAMDGPUAnnotateKernelFeaturesPass() { - return new AMDGPUAnnotateKernelFeatures(); -} diff --git a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def index 538b1b181f643..60d27a7fbef29 100644 --- a/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def +++ b/llvm/lib/Target/AMDGPU/AMDGPUPassRegistry.def @@ -149,9 +149,3 @@ DUMMY_MACHINE_FUNCTION_PASS("amdgpu-regbanklegalize", AMDGPURegBankLegalizePass( DUMMY_MACHINE_FUNCTION_PASS("amdgpu-regbank-combiner", AMDGPURegBankCombinerPass()) #undef DUMMY_MACHINE_FUNCTION_PASS - - -#define DUMMY_CGSCC_PASS(NAME, CREATE_PASS) -DUMMY_CGSCC_PASS("amdgpu-annotate-kernel-features", AMDGPUAnnotateKernelFeaturesPass()) - -#undef DUMMY_CGSCC_PASS diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp index 5b2e0558d5664..34dacd5f9209d 100644 --- a/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp +++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp @@ -515,7 +515,6 @@ extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() { initializeAMDGPUAlwaysInlinePass(*PR); initializeAMDGPUSwLowerLDSLegacyPass(*PR); initializeAMDGPUAttributorLegacyPass(*PR); - initializeAMDGPUAnnotateKernelFeaturesPass(*PR); initializeAMDGPUAnnotateUniformValuesLegacyPass(*PR); initializeAMDGPUArgumentUsageInfoPass(*PR); initializeAMDGPUAtomicOptimizerPass(*PR); @@ -1317,12 +1316,6 @@ void AMDGPUPassConfig::addIRPasses() { } void AMDGPUPassConfig::addCodeGenPrepare() { - if (TM->getTargetTriple().isAMDGCN()) { - // FIXME: This pass adds 2 hacky attributes that can be replaced with an - // analysis, and should be removed. - addPass(createAMDGPUAnnotateKernelFeaturesPass()); - } - if (TM->getTargetTriple().isAMDGCN() && EnableLowerKernelArguments) addPass(createAMDGPULowerKernelArgumentsPass()); diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp index 53f5c1efd14eb..d6153ce93b451 100644 --- a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp +++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp @@ -601,12 +601,6 @@ GCNUserSGPRUsageInfo::GCNUserSGPRUsageInfo(const Function &F, const CallingConv::ID CC = F.getCallingConv(); const bool IsKernel = CC == CallingConv::AMDGPU_KERNEL || CC == CallingConv::SPIR_KERNEL; - // FIXME: Should have analysis or something rather than attribute to detect - // calls. - const bool HasCalls = F.hasFnAttribute("amdgpu-calls"); - // FIXME: This attribute is a hack, we just need an analysis on the function - // to look for allocas. - const bool HasStackObjects = F.hasFnAttribute("amdgpu-stack-objects"); if (IsKernel && (!F.arg_empty() || ST.getImplicitArgNumBytes(F) != 0)) KernargSegmentPtr = true; @@ -629,12 +623,13 @@ GCNUserSGPRUsageInfo::GCNUserSGPRUsageInfo(const Function &F, DispatchID = true; } - // TODO: This could be refined a lot. The attribute is a poor way of - // detecting calls or stack objects that may require it before argument - // lowering. if (ST.hasFlatAddressSpace() && AMDGPU::isEntryFunctionCC(CC) && (IsAmdHsaOrMesa || ST.enableFlatScratch()) && - (HasCalls || HasStackObjects || ST.enableFlatScratch()) && + // FlatScratchInit cannot be true for graphics CC if enableFlatScratch() + // is false. + (ST.enableFlatScratch() || + (!AMDGPU::isGraphics(CC) && + !F.hasFnAttribute("amdgpu-no-flat-scratch-init"))) && !ST.flatScratchIsArchitected()) { FlatScratchInit = true; } diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_udec_wrap.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_udec_wrap.ll index d9be677a0e58d..aeb301939e986 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_udec_wrap.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_udec_wrap.ll @@ -20,11 +20,14 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i32(ptr addrspace(1) %out, ptr add ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, 42 ; CI-NEXT: s_mov_b32 m0, -1 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: ds_dec_rtn_u32 v2, v1, v0 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: flat_store_dword v[0:1], v2 ; CI-NEXT: s_endpgm @@ -35,11 +38,14 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i32(ptr addrspace(1) %out, ptr add ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, 42 ; VI-NEXT: s_mov_b32 m0, -1 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: ds_dec_rtn_u32 v2, v1, v0 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: s_endpgm @@ -97,11 +103,14 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i32_offset(ptr addrspace(1) %out, ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, 42 ; CI-NEXT: s_mov_b32 m0, -1 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: ds_dec_rtn_u32 v2, v1, v0 offset:16 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: flat_store_dword v[0:1], v2 ; CI-NEXT: s_endpgm @@ -112,11 +121,14 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i32_offset(ptr addrspace(1) %out, ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, 42 ; VI-NEXT: s_mov_b32 m0, -1 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: ds_dec_rtn_u32 v2, v1, v0 offset:16 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: s_endpgm @@ -287,6 +299,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32(ptr addrspace(1) %out, ptr ; CI-LABEL: global_atomic_dec_ret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 @@ -302,6 +317,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32(ptr addrspace(1) %out, ptr ; VI-LABEL: global_atomic_dec_ret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 @@ -359,6 +377,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset(ptr addrspace(1) %ou ; CI-LABEL: global_atomic_dec_ret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -376,6 +397,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset(ptr addrspace(1) %ou ; VI-LABEL: global_atomic_dec_ret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -436,6 +460,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset_system(ptr addrspace ; CI-LABEL: global_atomic_dec_ret_i32_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -453,6 +480,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset_system(ptr addrspace ; VI-LABEL: global_atomic_dec_ret_i32_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -513,6 +543,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32(ptr addrspace(1) %ptr) #1 ; CI-LABEL: global_atomic_dec_noret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -525,6 +558,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32(ptr addrspace(1) %ptr) #1 ; VI-LABEL: global_atomic_dec_noret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -575,6 +611,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32_offset(ptr addrspace(1) % ; CI-LABEL: global_atomic_dec_noret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -589,6 +628,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32_offset(ptr addrspace(1) % ; VI-LABEL: global_atomic_dec_noret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -642,6 +684,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32_offset_system(ptr addrspa ; CI-LABEL: global_atomic_dec_noret_i32_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -656,6 +701,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32_offset_system(ptr addrspa ; VI-LABEL: global_atomic_dec_noret_i32_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -710,7 +758,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset_addr64(ptr addrspace ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; CI-NEXT: v_mov_b32_e32 v3, 42 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -718,6 +768,7 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset_addr64(ptr addrspace ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_add_i32_e32 v0, vcc, 20, v0 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: v_mov_b32_e32 v3, 42 ; CI-NEXT: flat_atomic_dec v3, v[0:1], v3 glc ; CI-NEXT: s_waitcnt vmcnt(0) ; CI-NEXT: buffer_wbinvl1_vol @@ -732,7 +783,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset_addr64(ptr addrspace ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; VI-NEXT: v_mov_b32_e32 v3, 42 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -740,6 +793,7 @@ define amdgpu_kernel void @global_atomic_dec_ret_i32_offset_addr64(ptr addrspace ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_add_u32_e32 v0, vcc, 20, v0 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: v_mov_b32_e32 v3, 42 ; VI-NEXT: flat_atomic_dec v3, v[0:1], v3 glc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: buffer_wbinvl1_vol @@ -802,6 +856,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32_offset_addr64(ptr addrspa ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -819,6 +876,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i32_offset_addr64(ptr addrspa ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -878,6 +938,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32(ptr %out, ptr %ptr) #1 { ; CI-LABEL: flat_atomic_dec_ret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 @@ -893,6 +956,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32(ptr %out, ptr %ptr) #1 { ; VI-LABEL: flat_atomic_dec_ret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 @@ -908,6 +974,8 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32(ptr %out, ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_dec_ret_i32: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -922,6 +990,10 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32(ptr %out, ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_dec_ret_i32: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -958,6 +1030,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset(ptr %out, ptr %ptr) #1 ; CI-LABEL: flat_atomic_dec_ret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -975,6 +1050,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset(ptr %out, ptr %ptr) #1 ; VI-LABEL: flat_atomic_dec_ret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -992,6 +1070,8 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset(ptr %out, ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_dec_ret_i32_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -1006,6 +1086,10 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset(ptr %out, ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_dec_ret_i32_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -1045,6 +1129,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_system(ptr %out, ptr % ; CI-LABEL: flat_atomic_dec_ret_i32_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -1062,6 +1149,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_system(ptr %out, ptr % ; VI-LABEL: flat_atomic_dec_ret_i32_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -1079,6 +1169,8 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_system(ptr %out, ptr % ; GFX9-LABEL: flat_atomic_dec_ret_i32_offset_system: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -1093,6 +1185,10 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_system(ptr %out, ptr % ; ; GFX10-LABEL: flat_atomic_dec_ret_i32_offset_system: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -1132,6 +1228,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_dec_noret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -1144,6 +1243,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_dec_noret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -1156,6 +1258,8 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_dec_noret_i32: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 @@ -1167,6 +1271,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_dec_noret_i32: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -1199,6 +1307,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_dec_noret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -1213,6 +1324,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_dec_noret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -1227,6 +1341,8 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_dec_noret_i32_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 @@ -1238,6 +1354,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_dec_noret_i32_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -1273,6 +1393,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_system(ptr %ptr) #1 ; CI-LABEL: flat_atomic_dec_noret_i32_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -1287,6 +1410,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_system(ptr %ptr) #1 ; VI-LABEL: flat_atomic_dec_noret_i32_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -1301,6 +1427,8 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_system(ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_dec_noret_i32_offset_system: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 @@ -1312,6 +1440,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_system(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_dec_noret_i32_offset_system: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -1348,7 +1480,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_addr64(ptr %out, ptr % ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; CI-NEXT: v_mov_b32_e32 v3, 42 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1356,6 +1490,7 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_addr64(ptr %out, ptr % ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_add_i32_e32 v0, vcc, 20, v0 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: v_mov_b32_e32 v3, 42 ; CI-NEXT: flat_atomic_dec v3, v[0:1], v3 glc ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) ; CI-NEXT: buffer_wbinvl1_vol @@ -1370,7 +1505,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_addr64(ptr %out, ptr % ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; VI-NEXT: v_mov_b32_e32 v3, 42 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1378,6 +1515,7 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_addr64(ptr %out, ptr % ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_add_u32_e32 v0, vcc, 20, v0 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: v_mov_b32_e32 v3, 42 ; VI-NEXT: flat_atomic_dec v3, v[0:1], v3 glc ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) ; VI-NEXT: buffer_wbinvl1_vol @@ -1392,6 +1530,8 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_addr64(ptr %out, ptr % ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v3, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -1410,6 +1550,10 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i32_offset_addr64(ptr %out, ptr % ; ; GFX10-LABEL: flat_atomic_dec_ret_i32_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v2, 2, v0 ; GFX10-NEXT: v_mov_b32_e32 v3, 42 @@ -1466,6 +1610,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_addr64(ptr %ptr) #1 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -1483,6 +1630,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_addr64(ptr %ptr) #1 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -1500,6 +1650,8 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_addr64(ptr %ptr) #1 ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 ; GFX9-NEXT: v_mov_b32_e32 v1, s1 @@ -1513,6 +1665,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i32_offset_addr64(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_dec_noret_i32_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v2, 2, v0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -1559,10 +1715,13 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64(ptr %out, ptr %ptr) #1 { ; CI-LABEL: flat_atomic_dec_ret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1580,10 +1739,13 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64(ptr %out, ptr %ptr) #1 { ; VI-LABEL: flat_atomic_dec_ret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1601,7 +1763,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64(ptr %out, ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_dec_ret_i64: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v2, s2 @@ -1616,6 +1780,10 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64(ptr %out, ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_dec_ret_i64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -1654,12 +1822,15 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset(ptr %out, ptr %ptr) #1 ; CI-LABEL: flat_atomic_dec_ret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 32 ; CI-NEXT: s_addc_u32 s3, s3, 0 ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1677,12 +1848,15 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset(ptr %out, ptr %ptr) #1 ; VI-LABEL: flat_atomic_dec_ret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 32 ; VI-NEXT: s_addc_u32 s3, s3, 0 ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1700,7 +1874,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset(ptr %out, ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_dec_ret_i64_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v2, s2 @@ -1715,6 +1891,10 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset(ptr %out, ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_dec_ret_i64_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -1756,10 +1936,13 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_dec_noret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1769,10 +1952,13 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_dec_noret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1782,7 +1968,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_dec_noret_i64: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v3, s1 @@ -1794,6 +1982,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_dec_noret_i64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -1828,12 +2020,15 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_dec_noret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1843,12 +2038,15 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_dec_noret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1858,7 +2056,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_dec_noret_i64_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v3, s1 @@ -1870,6 +2070,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_dec_noret_i64_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -1907,12 +2111,15 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_system(ptr %ptr) #1 ; CI-LABEL: flat_atomic_dec_noret_i64_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1922,12 +2129,15 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_system(ptr %ptr) #1 ; VI-LABEL: flat_atomic_dec_noret_i64_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1937,7 +2147,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_system(ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_dec_noret_i64_offset_system: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v3, s1 @@ -1949,6 +2161,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_system(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_dec_noret_i64_offset_system: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -1987,6 +2203,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset_addr64(ptr %out, ptr % ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2013,6 +2232,9 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset_addr64(ptr %out, ptr % ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2039,12 +2261,14 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset_addr64(ptr %out, ptr % ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v5, 3, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v1, 42 -; GFX9-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v4, s3 ; GFX9-NEXT: v_mov_b32_e32 v3, s2 ; GFX9-NEXT: v_add_co_u32_e32 v3, vcc, v3, v5 +; GFX9-NEXT: v_mov_b32_e32 v2, 0 ; GFX9-NEXT: v_addc_co_u32_e32 v4, vcc, 0, v4, vcc ; GFX9-NEXT: flat_atomic_dec_x2 v[0:1], v[3:4], v[1:2] offset:40 glc ; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -2058,6 +2282,10 @@ define amdgpu_kernel void @flat_atomic_dec_ret_i64_offset_addr64(ptr %out, ptr % ; ; GFX10-LABEL: flat_atomic_dec_ret_i64_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v4, 3, v0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2116,6 +2344,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_addr64(ptr %ptr) #1 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -2134,6 +2365,9 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_addr64(ptr %ptr) #1 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -2152,12 +2386,14 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_addr64(ptr %ptr) #1 ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v1, 42 -; GFX9-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v4, s1 ; GFX9-NEXT: v_mov_b32_e32 v3, s0 ; GFX9-NEXT: v_add_co_u32_e32 v3, vcc, v3, v0 +; GFX9-NEXT: v_mov_b32_e32 v2, 0 ; GFX9-NEXT: v_addc_co_u32_e32 v4, vcc, 0, v4, vcc ; GFX9-NEXT: flat_atomic_dec_x2 v[3:4], v[1:2] offset:40 ; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -2166,6 +2402,10 @@ define amdgpu_kernel void @flat_atomic_dec_noret_i64_offset_addr64(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_dec_noret_i64_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v2, 3, v0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2219,8 +2459,11 @@ define amdgpu_kernel void @atomic_dec_shl_base_lds_0(ptr addrspace(1) %out, ptr ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: ds_dec_rtn_u32 v2, v1, v2 offset:8 ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_add_i32_e32 v3, vcc, 2, v0 ; CI-NEXT: v_mov_b32_e32 v0, s2 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: flat_store_dword v[0:1], v3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -2237,8 +2480,11 @@ define amdgpu_kernel void @atomic_dec_shl_base_lds_0(ptr addrspace(1) %out, ptr ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: ds_dec_rtn_u32 v2, v1, v2 offset:8 ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_add_u32_e32 v3, vcc, 2, v0 ; VI-NEXT: v_mov_b32_e32 v0, s2 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: flat_store_dword v[0:1], v3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -2312,7 +2558,10 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i64(ptr addrspace(1) %out, ptr add ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: ds_dec_rtn_u64 v[0:1], v2, v[0:1] ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CI-NEXT: s_endpgm @@ -2328,7 +2577,10 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i64(ptr addrspace(1) %out, ptr add ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: ds_dec_rtn_u64 v[0:1], v2, v[0:1] ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; VI-NEXT: s_endpgm @@ -2394,7 +2646,10 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i64_offset(ptr addrspace(1) %out, ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: ds_dec_rtn_u64 v[0:1], v2, v[0:1] offset:32 ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CI-NEXT: s_endpgm @@ -2410,7 +2665,10 @@ define amdgpu_kernel void @lds_atomic_dec_ret_i64_offset(ptr addrspace(1) %out, ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: ds_dec_rtn_u64 v[0:1], v2, v[0:1] offset:32 ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; VI-NEXT: s_endpgm @@ -2594,10 +2852,13 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64(ptr addrspace(1) %out, ptr ; CI-LABEL: global_atomic_dec_ret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) @@ -2610,10 +2871,13 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64(ptr addrspace(1) %out, ptr ; VI-LABEL: global_atomic_dec_ret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) @@ -2671,12 +2935,15 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64_offset(ptr addrspace(1) %ou ; CI-LABEL: global_atomic_dec_ret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 32 ; CI-NEXT: s_addc_u32 s3, s3, 0 ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) @@ -2689,12 +2956,15 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64_offset(ptr addrspace(1) %ou ; VI-LABEL: global_atomic_dec_ret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 32 ; VI-NEXT: s_addc_u32 s3, s3, 0 ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) @@ -2753,12 +3023,15 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64_offset_system(ptr addrspace ; CI-LABEL: global_atomic_dec_ret_i64_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 32 ; CI-NEXT: s_addc_u32 s3, s3, 0 ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) @@ -2771,12 +3044,15 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64_offset_system(ptr addrspace ; VI-LABEL: global_atomic_dec_ret_i64_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 32 ; VI-NEXT: s_addc_u32 s3, s3, 0 ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_dec_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) @@ -2835,10 +3111,13 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64(ptr addrspace(1) %ptr) #1 ; CI-LABEL: global_atomic_dec_noret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) @@ -2848,10 +3127,13 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64(ptr addrspace(1) %ptr) #1 ; VI-LABEL: global_atomic_dec_noret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) @@ -2902,12 +3184,15 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64_offset(ptr addrspace(1) % ; CI-LABEL: global_atomic_dec_noret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) @@ -2917,12 +3202,15 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64_offset(ptr addrspace(1) % ; VI-LABEL: global_atomic_dec_noret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) @@ -2974,12 +3262,15 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64_offset_system(ptr addrspa ; CI-LABEL: global_atomic_dec_noret_i64_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) @@ -2989,12 +3280,15 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64_offset_system(ptr addrspa ; VI-LABEL: global_atomic_dec_noret_i64_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_dec_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) @@ -3047,6 +3341,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64_offset_addr64(ptr addrspace ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -3070,6 +3367,9 @@ define amdgpu_kernel void @global_atomic_dec_ret_i64_offset_addr64(ptr addrspace ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -3144,6 +3444,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64_offset_addr64(ptr addrspa ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -3162,6 +3465,9 @@ define amdgpu_kernel void @global_atomic_dec_noret_i64_offset_addr64(ptr addrspa ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -3232,7 +3538,10 @@ define amdgpu_kernel void @atomic_dec_shl_base_lds_0_i64(ptr addrspace(1) %out, ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: ds_dec_rtn_u64 v[1:2], v3, v[1:2] offset:16 ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v4, s3 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_add_i32_e32 v0, vcc, 2, v0 ; CI-NEXT: v_mov_b32_e32 v3, s2 ; CI-NEXT: flat_store_dword v[3:4], v0 @@ -3251,7 +3560,10 @@ define amdgpu_kernel void @atomic_dec_shl_base_lds_0_i64(ptr addrspace(1) %out, ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: ds_dec_rtn_u64 v[1:2], v3, v[1:2] offset:16 ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v4, s3 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_add_u32_e32 v0, vcc, 2, v0 ; VI-NEXT: v_mov_b32_e32 v3, s2 ; VI-NEXT: flat_store_dword v[3:4], v0 diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_uinc_wrap.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_uinc_wrap.ll index 92a7de9aaefd2..1d401a4ee33d8 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_uinc_wrap.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/atomicrmw_uinc_wrap.ll @@ -21,11 +21,14 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i32(ptr addrspace(1) %out, ptr add ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, 42 ; CI-NEXT: s_mov_b32 m0, -1 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: ds_inc_rtn_u32 v2, v1, v0 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: flat_store_dword v[0:1], v2 ; CI-NEXT: s_endpgm @@ -36,11 +39,14 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i32(ptr addrspace(1) %out, ptr add ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, 42 ; VI-NEXT: s_mov_b32 m0, -1 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: ds_inc_rtn_u32 v2, v1, v0 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: s_endpgm @@ -110,11 +116,14 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i32_offset(ptr addrspace(1) %out, ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, 42 ; CI-NEXT: s_mov_b32 m0, -1 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: ds_inc_rtn_u32 v2, v1, v0 offset:16 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: flat_store_dword v[0:1], v2 ; CI-NEXT: s_endpgm @@ -125,11 +134,14 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i32_offset(ptr addrspace(1) %out, ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, 42 ; VI-NEXT: s_mov_b32 m0, -1 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: ds_inc_rtn_u32 v2, v1, v0 offset:16 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: s_endpgm @@ -332,6 +344,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32(ptr addrspace(1) %out, ptr ; CI-LABEL: global_atomic_inc_ret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 @@ -347,6 +362,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32(ptr addrspace(1) %out, ptr ; VI-LABEL: global_atomic_inc_ret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 @@ -415,6 +433,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset(ptr addrspace(1) %ou ; CI-LABEL: global_atomic_inc_ret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -432,6 +453,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset(ptr addrspace(1) %ou ; VI-LABEL: global_atomic_inc_ret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -503,6 +527,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset_sistem(ptr addrspace ; CI-LABEL: global_atomic_inc_ret_i32_offset_sistem: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -520,6 +547,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset_sistem(ptr addrspace ; VI-LABEL: global_atomic_inc_ret_i32_offset_sistem: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -592,6 +622,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32(ptr addrspace(1) %ptr) #1 ; CI-LABEL: global_atomic_inc_noret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -604,6 +637,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32(ptr addrspace(1) %ptr) #1 ; VI-LABEL: global_atomic_inc_noret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -664,6 +700,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32_offset(ptr addrspace(1) % ; CI-LABEL: global_atomic_inc_noret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -678,6 +717,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32_offset(ptr addrspace(1) % ; VI-LABEL: global_atomic_inc_noret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -741,6 +783,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32_offset_system(ptr addrspa ; CI-LABEL: global_atomic_inc_noret_i32_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -755,6 +800,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32_offset_system(ptr addrspa ; VI-LABEL: global_atomic_inc_noret_i32_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -820,7 +868,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset_addr64(ptr addrspace ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; CI-NEXT: v_mov_b32_e32 v3, 42 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -828,6 +878,7 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset_addr64(ptr addrspace ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_add_i32_e32 v0, vcc, 20, v0 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: v_mov_b32_e32 v3, 42 ; CI-NEXT: flat_atomic_inc v3, v[0:1], v3 glc ; CI-NEXT: s_waitcnt vmcnt(0) ; CI-NEXT: buffer_wbinvl1_vol @@ -842,7 +893,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset_addr64(ptr addrspace ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; VI-NEXT: v_mov_b32_e32 v3, 42 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -850,6 +903,7 @@ define amdgpu_kernel void @global_atomic_inc_ret_i32_offset_addr64(ptr addrspace ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_add_u32_e32 v0, vcc, 20, v0 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: v_mov_b32_e32 v3, 42 ; VI-NEXT: flat_atomic_inc v3, v[0:1], v3 glc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: buffer_wbinvl1_vol @@ -925,6 +979,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32_offset_addr64(ptr addrspa ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -942,6 +999,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i32_offset_addr64(ptr addrspa ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -1019,8 +1079,11 @@ define amdgpu_kernel void @atomic_inc_shl_base_lds_0_i32(ptr addrspace(1) %out, ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: ds_inc_rtn_u32 v2, v1, v2 offset:8 ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_add_i32_e32 v3, vcc, 2, v0 ; CI-NEXT: v_mov_b32_e32 v0, s2 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: flat_store_dword v[0:1], v3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -1037,8 +1100,11 @@ define amdgpu_kernel void @atomic_inc_shl_base_lds_0_i32(ptr addrspace(1) %out, ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: ds_inc_rtn_u32 v2, v1, v2 offset:8 ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_add_u32_e32 v3, vcc, 2, v0 ; VI-NEXT: v_mov_b32_e32 v0, s2 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: flat_store_dword v[0:1], v3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -1129,7 +1195,10 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i64(ptr addrspace(1) %out, ptr add ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: ds_inc_rtn_u64 v[0:1], v2, v[0:1] ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CI-NEXT: s_endpgm @@ -1145,7 +1214,10 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i64(ptr addrspace(1) %out, ptr add ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: ds_inc_rtn_u64 v[0:1], v2, v[0:1] ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; VI-NEXT: s_endpgm @@ -1224,7 +1296,10 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i64_offset(ptr addrspace(1) %out, ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: ds_inc_rtn_u64 v[0:1], v2, v[0:1] offset:32 ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CI-NEXT: s_endpgm @@ -1240,7 +1315,10 @@ define amdgpu_kernel void @lds_atomic_inc_ret_i64_offset(ptr addrspace(1) %out, ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: ds_inc_rtn_u64 v[0:1], v2, v[0:1] offset:32 ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; VI-NEXT: s_endpgm @@ -1459,10 +1537,13 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64(ptr addrspace(1) %out, ptr ; CI-LABEL: global_atomic_inc_ret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) @@ -1475,10 +1556,13 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64(ptr addrspace(1) %out, ptr ; VI-LABEL: global_atomic_inc_ret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) @@ -1548,12 +1632,15 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64_offset(ptr addrspace(1) %ou ; CI-LABEL: global_atomic_inc_ret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 32 ; CI-NEXT: s_addc_u32 s3, s3, 0 ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) @@ -1566,12 +1653,15 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64_offset(ptr addrspace(1) %ou ; VI-LABEL: global_atomic_inc_ret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 32 ; VI-NEXT: s_addc_u32 s3, s3, 0 ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) @@ -1642,12 +1732,15 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64_offset_system(ptr addrspace ; CI-LABEL: global_atomic_inc_ret_i64_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 32 ; CI-NEXT: s_addc_u32 s3, s3, 0 ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) @@ -1660,12 +1753,15 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64_offset_system(ptr addrspace ; VI-LABEL: global_atomic_inc_ret_i64_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 32 ; VI-NEXT: s_addc_u32 s3, s3, 0 ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) @@ -1737,10 +1833,13 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64(ptr addrspace(1) %ptr) #1 ; CI-LABEL: global_atomic_inc_noret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) @@ -1750,10 +1849,13 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64(ptr addrspace(1) %ptr) #1 ; VI-LABEL: global_atomic_inc_noret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) @@ -1815,12 +1917,15 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64_offset(ptr addrspace(1) % ; CI-LABEL: global_atomic_inc_noret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) @@ -1830,12 +1935,15 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64_offset(ptr addrspace(1) % ; VI-LABEL: global_atomic_inc_noret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) @@ -1898,12 +2006,15 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64_offset_system(ptr addrspa ; CI-LABEL: global_atomic_inc_noret_i64_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) @@ -1913,12 +2024,15 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64_offset_system(ptr addrspa ; VI-LABEL: global_atomic_inc_noret_i64_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) @@ -1983,6 +2097,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64_offset_addr64(ptr addrspace ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2006,6 +2123,9 @@ define amdgpu_kernel void @global_atomic_inc_ret_i64_offset_addr64(ptr addrspace ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2094,6 +2214,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64_offset_addr64(ptr addrspa ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -2112,6 +2235,9 @@ define amdgpu_kernel void @global_atomic_inc_noret_i64_offset_addr64(ptr addrspa ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -2188,6 +2314,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32(ptr %out, ptr %ptr) #1 { ; CI-LABEL: flat_atomic_inc_ret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 @@ -2203,6 +2332,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32(ptr %out, ptr %ptr) #1 { ; VI-LABEL: flat_atomic_inc_ret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 @@ -2218,6 +2350,8 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32(ptr %out, ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_inc_ret_i32: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -2232,6 +2366,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32(ptr %out, ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_inc_ret_i32: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2281,6 +2419,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset(ptr %out, ptr %ptr) #1 ; CI-LABEL: flat_atomic_inc_ret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -2298,6 +2439,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset(ptr %out, ptr %ptr) #1 ; VI-LABEL: flat_atomic_inc_ret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -2315,6 +2459,8 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset(ptr %out, ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_inc_ret_i32_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -2329,6 +2475,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset(ptr %out, ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_inc_ret_i32_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2381,6 +2531,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_system(ptr %out, ptr % ; CI-LABEL: flat_atomic_inc_ret_i32_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 16 @@ -2398,6 +2551,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_system(ptr %out, ptr % ; VI-LABEL: flat_atomic_inc_ret_i32_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 16 @@ -2415,6 +2571,8 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_system(ptr %out, ptr % ; GFX9-LABEL: flat_atomic_inc_ret_i32_offset_system: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -2429,6 +2587,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_system(ptr %out, ptr % ; ; GFX10-LABEL: flat_atomic_inc_ret_i32_offset_system: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2482,6 +2644,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_inc_noret_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -2494,6 +2659,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_inc_noret_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -2506,6 +2674,8 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_inc_noret_i32: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 @@ -2517,6 +2687,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_inc_noret_i32: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2560,6 +2734,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_inc_noret_i32_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -2574,6 +2751,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_inc_noret_i32_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -2588,6 +2768,8 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_inc_noret_i32_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 @@ -2599,6 +2781,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_inc_noret_i32_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2645,6 +2831,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_system(ptr %ptr) #1 ; CI-LABEL: flat_atomic_inc_noret_i32_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: v_mov_b32_e32 v2, 42 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 16 @@ -2659,6 +2848,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_system(ptr %ptr) #1 ; VI-LABEL: flat_atomic_inc_noret_i32_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: v_mov_b32_e32 v2, 42 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 16 @@ -2673,6 +2865,8 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_system(ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_inc_noret_i32_offset_system: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v2, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 @@ -2684,6 +2878,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_system(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_inc_noret_i32_offset_system: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v2, 42 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2732,7 +2930,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_addr64(ptr %out, ptr % ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; CI-NEXT: v_mov_b32_e32 v3, 42 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2740,6 +2940,7 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_addr64(ptr %out, ptr % ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_add_i32_e32 v0, vcc, 20, v0 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: v_mov_b32_e32 v3, 42 ; CI-NEXT: flat_atomic_inc v3, v[0:1], v3 glc ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) ; CI-NEXT: buffer_wbinvl1_vol @@ -2754,7 +2955,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_addr64(ptr %out, ptr % ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 -; VI-NEXT: v_mov_b32_e32 v3, 42 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2762,6 +2965,7 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_addr64(ptr %out, ptr % ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_add_u32_e32 v0, vcc, 20, v0 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: v_mov_b32_e32 v3, 42 ; VI-NEXT: flat_atomic_inc v3, v[0:1], v3 glc ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) ; VI-NEXT: buffer_wbinvl1_vol @@ -2776,6 +2980,8 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_addr64(ptr %out, ptr % ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v3, 42 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 @@ -2794,6 +3000,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i32_offset_addr64(ptr %out, ptr % ; ; GFX10-LABEL: flat_atomic_inc_ret_i32_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v2, 2, v0 ; GFX10-NEXT: v_mov_b32_e32 v3, 42 @@ -2871,6 +3081,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_addr64(ptr %ptr) #1 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -2888,6 +3101,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_addr64(ptr %ptr) #1 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -2905,6 +3121,8 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_addr64(ptr %ptr) #1 ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s0 ; GFX9-NEXT: v_mov_b32_e32 v1, s1 @@ -2918,6 +3136,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i32_offset_addr64(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_inc_noret_i32_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v2, 2, v0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -2988,7 +3210,10 @@ define amdgpu_kernel void @atomic_inc_shl_base_lds_0_i64(ptr addrspace(1) %out, ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: ds_inc_rtn_u64 v[1:2], v3, v[1:2] offset:16 ; CI-NEXT: s_waitcnt lgkmcnt(0) +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v4, s3 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_add_i32_e32 v0, vcc, 2, v0 ; CI-NEXT: v_mov_b32_e32 v3, s2 ; CI-NEXT: flat_store_dword v[3:4], v0 @@ -3007,7 +3232,10 @@ define amdgpu_kernel void @atomic_inc_shl_base_lds_0_i64(ptr addrspace(1) %out, ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: ds_inc_rtn_u64 v[1:2], v3, v[1:2] offset:16 ; VI-NEXT: s_waitcnt lgkmcnt(0) +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v4, s3 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_add_u32_e32 v0, vcc, 2, v0 ; VI-NEXT: v_mov_b32_e32 v3, s2 ; VI-NEXT: flat_store_dword v[3:4], v0 @@ -3097,10 +3325,13 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64(ptr %out, ptr %ptr) #1 { ; CI-LABEL: flat_atomic_inc_ret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3118,10 +3349,13 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64(ptr %out, ptr %ptr) #1 { ; VI-LABEL: flat_atomic_inc_ret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3139,7 +3373,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64(ptr %out, ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_inc_ret_i64: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v2, s2 @@ -3154,6 +3390,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64(ptr %out, ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_inc_ret_i64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -3206,12 +3446,15 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset(ptr %out, ptr %ptr) #1 ; CI-LABEL: flat_atomic_inc_ret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 32 ; CI-NEXT: s_addc_u32 s3, s3, 0 ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3229,12 +3472,15 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset(ptr %out, ptr %ptr) #1 ; VI-LABEL: flat_atomic_inc_ret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 32 ; VI-NEXT: s_addc_u32 s3, s3, 0 ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3252,7 +3498,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset(ptr %out, ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_inc_ret_i64_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v2, s2 @@ -3267,6 +3515,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset(ptr %out, ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_inc_ret_i64_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -3322,12 +3574,15 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_system(ptr %out, ptr % ; CI-LABEL: flat_atomic_inc_ret_i64_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s2, s2, 32 ; CI-NEXT: s_addc_u32 s3, s3, 0 ; CI-NEXT: v_mov_b32_e32 v2, s2 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3345,12 +3600,15 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_system(ptr %out, ptr % ; VI-LABEL: flat_atomic_inc_ret_i64_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s2, s2, 32 ; VI-NEXT: s_addc_u32 s3, s3, 0 ; VI-NEXT: v_mov_b32_e32 v2, s2 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_atomic_inc_x2 v[0:1], v[2:3], v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3368,7 +3626,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_system(ptr %out, ptr % ; GFX9-LABEL: flat_atomic_inc_ret_i64_offset_system: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v2, s2 @@ -3383,6 +3643,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_system(ptr %out, ptr % ; ; GFX10-LABEL: flat_atomic_inc_ret_i64_offset_system: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -3439,10 +3703,13 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_inc_noret_i64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3452,10 +3719,13 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_inc_noret_i64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3465,7 +3735,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_inc_noret_i64: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v3, s1 @@ -3477,6 +3749,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_inc_noret_i64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -3523,12 +3799,15 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset(ptr %ptr) #1 { ; CI-LABEL: flat_atomic_inc_noret_i64_offset: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3538,12 +3817,15 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset(ptr %ptr) #1 { ; VI-LABEL: flat_atomic_inc_noret_i64_offset: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3553,7 +3835,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset(ptr %ptr) #1 { ; GFX9-LABEL: flat_atomic_inc_noret_i64_offset: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v3, s1 @@ -3565,6 +3849,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset(ptr %ptr) #1 { ; ; GFX10-LABEL: flat_atomic_inc_noret_i64_offset: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -3614,12 +3902,15 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_system(ptr %ptr) #1 ; CI-LABEL: flat_atomic_inc_noret_i64_offset_system: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v0, 42 -; CI-NEXT: v_mov_b32_e32 v1, 0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s0, s0, 32 ; CI-NEXT: s_addc_u32 s1, s1, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v1, 0 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; CI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3629,12 +3920,15 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_system(ptr %ptr) #1 ; VI-LABEL: flat_atomic_inc_noret_i64_offset_system: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v0, 42 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s0, s0, 32 ; VI-NEXT: s_addc_u32 s1, s1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_atomic_inc_x2 v[2:3], v[0:1] ; VI-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3644,7 +3938,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_system(ptr %ptr) #1 ; GFX9-LABEL: flat_atomic_inc_noret_i64_offset_system: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v0, 42 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: v_mov_b32_e32 v1, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v3, s1 @@ -3656,6 +3952,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_system(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_inc_noret_i64_offset_system: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_mov_b32_e32 v0, 42 ; GFX10-NEXT: v_mov_b32_e32 v1, 0 @@ -3707,6 +4007,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_addr64(ptr %out, ptr % ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -3733,6 +4036,9 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_addr64(ptr %out, ptr % ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -3759,12 +4065,14 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_addr64(ptr %out, ptr % ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v5, 3, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v1, 42 -; GFX9-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v4, s3 ; GFX9-NEXT: v_mov_b32_e32 v3, s2 ; GFX9-NEXT: v_add_co_u32_e32 v3, vcc, v3, v5 +; GFX9-NEXT: v_mov_b32_e32 v2, 0 ; GFX9-NEXT: v_addc_co_u32_e32 v4, vcc, 0, v4, vcc ; GFX9-NEXT: flat_atomic_inc_x2 v[0:1], v[3:4], v[1:2] offset:40 glc ; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3778,6 +4086,10 @@ define amdgpu_kernel void @flat_atomic_inc_ret_i64_offset_addr64(ptr %out, ptr % ; ; GFX10-LABEL: flat_atomic_inc_ret_i64_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v4, 3, v0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -3858,6 +4170,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_addr64(ptr %ptr) #1 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -3876,6 +4191,9 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_addr64(ptr %ptr) #1 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -3894,12 +4212,14 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_addr64(ptr %ptr) #1 ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX9-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 ; GFX9-NEXT: v_mov_b32_e32 v1, 42 -; GFX9-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v4, s1 ; GFX9-NEXT: v_mov_b32_e32 v3, s0 ; GFX9-NEXT: v_add_co_u32_e32 v3, vcc, v3, v0 +; GFX9-NEXT: v_mov_b32_e32 v2, 0 ; GFX9-NEXT: v_addc_co_u32_e32 v4, vcc, 0, v4, vcc ; GFX9-NEXT: flat_atomic_inc_x2 v[3:4], v[1:2] offset:40 ; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -3908,6 +4228,10 @@ define amdgpu_kernel void @flat_atomic_inc_noret_i64_offset_addr64(ptr %ptr) #1 ; ; GFX10-LABEL: flat_atomic_inc_noret_i64_offset_addr64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: v_lshlrev_b32_e32 v2, 3, v0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) @@ -3975,6 +4299,7 @@ define amdgpu_kernel void @nocse_lds_atomic_inc_ret_i32(ptr addrspace(1) %out0, ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, 42 ; CI-NEXT: s_mov_b32 m0, -1 +; CI-NEXT: s_add_i32 s12, s12, s17 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s4 ; CI-NEXT: ds_inc_rtn_u32 v2, v1, v0 @@ -3982,6 +4307,8 @@ define amdgpu_kernel void @nocse_lds_atomic_inc_ret_i32(ptr addrspace(1) %out0, ; CI-NEXT: ds_inc_rtn_u32 v3, v1, v0 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: flat_store_dword v[0:1], v2 ; CI-NEXT: v_mov_b32_e32 v0, s2 @@ -3995,6 +4322,7 @@ define amdgpu_kernel void @nocse_lds_atomic_inc_ret_i32(ptr addrspace(1) %out0, ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, 42 ; VI-NEXT: s_mov_b32 m0, -1 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s4 ; VI-NEXT: ds_inc_rtn_u32 v2, v1, v0 @@ -4002,6 +4330,8 @@ define amdgpu_kernel void @nocse_lds_atomic_inc_ret_i32(ptr addrspace(1) %out0, ; VI-NEXT: ds_inc_rtn_u32 v3, v1, v0 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: v_mov_b32_e32 v0, s2 diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll index 31a229a908142..9ef16aef0dd16 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll @@ -3016,7 +3016,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; GPRIDX-NEXT: kernel_code_entry_byte_offset = 256 ; GPRIDX-NEXT: kernel_code_prefetch_byte_size = 0 ; GPRIDX-NEXT: granulated_workitem_vgpr_count = 0 -; GPRIDX-NEXT: granulated_wavefront_sgpr_count = 1 +; GPRIDX-NEXT: granulated_wavefront_sgpr_count = 2 ; GPRIDX-NEXT: priority = 0 ; GPRIDX-NEXT: float_mode = 240 ; GPRIDX-NEXT: priv = 0 @@ -3027,7 +3027,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; GPRIDX-NEXT: enable_mem_ordered = 0 ; GPRIDX-NEXT: enable_fwd_progress = 0 ; GPRIDX-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; GPRIDX-NEXT: user_sgpr_count = 12 +; GPRIDX-NEXT: user_sgpr_count = 14 ; GPRIDX-NEXT: enable_trap_handler = 0 ; GPRIDX-NEXT: enable_sgpr_workgroup_id_x = 1 ; GPRIDX-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -3042,7 +3042,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; GPRIDX-NEXT: enable_sgpr_queue_ptr = 1 ; GPRIDX-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; GPRIDX-NEXT: enable_sgpr_dispatch_id = 1 -; GPRIDX-NEXT: enable_sgpr_flat_scratch_init = 0 +; GPRIDX-NEXT: enable_sgpr_flat_scratch_init = 1 ; GPRIDX-NEXT: enable_sgpr_private_segment_size = 0 ; GPRIDX-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; GPRIDX-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -3059,7 +3059,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; GPRIDX-NEXT: gds_segment_byte_size = 0 ; GPRIDX-NEXT: kernarg_segment_byte_size = 28 ; GPRIDX-NEXT: workgroup_fbarrier_count = 0 -; GPRIDX-NEXT: wavefront_sgpr_count = 15 +; GPRIDX-NEXT: wavefront_sgpr_count = 17 ; GPRIDX-NEXT: workitem_vgpr_count = 3 ; GPRIDX-NEXT: reserved_vgpr_first = 0 ; GPRIDX-NEXT: reserved_vgpr_count = 0 @@ -3107,7 +3107,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; MOVREL-NEXT: kernel_code_entry_byte_offset = 256 ; MOVREL-NEXT: kernel_code_prefetch_byte_size = 0 ; MOVREL-NEXT: granulated_workitem_vgpr_count = 0 -; MOVREL-NEXT: granulated_wavefront_sgpr_count = 1 +; MOVREL-NEXT: granulated_wavefront_sgpr_count = 2 ; MOVREL-NEXT: priority = 0 ; MOVREL-NEXT: float_mode = 240 ; MOVREL-NEXT: priv = 0 @@ -3118,7 +3118,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; MOVREL-NEXT: enable_mem_ordered = 0 ; MOVREL-NEXT: enable_fwd_progress = 0 ; MOVREL-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; MOVREL-NEXT: user_sgpr_count = 12 +; MOVREL-NEXT: user_sgpr_count = 14 ; MOVREL-NEXT: enable_trap_handler = 0 ; MOVREL-NEXT: enable_sgpr_workgroup_id_x = 1 ; MOVREL-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -3133,7 +3133,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; MOVREL-NEXT: enable_sgpr_queue_ptr = 1 ; MOVREL-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; MOVREL-NEXT: enable_sgpr_dispatch_id = 1 -; MOVREL-NEXT: enable_sgpr_flat_scratch_init = 0 +; MOVREL-NEXT: enable_sgpr_flat_scratch_init = 1 ; MOVREL-NEXT: enable_sgpr_private_segment_size = 0 ; MOVREL-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; MOVREL-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -3150,7 +3150,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; MOVREL-NEXT: gds_segment_byte_size = 0 ; MOVREL-NEXT: kernarg_segment_byte_size = 28 ; MOVREL-NEXT: workgroup_fbarrier_count = 0 -; MOVREL-NEXT: wavefront_sgpr_count = 10 +; MOVREL-NEXT: wavefront_sgpr_count = 24 ; MOVREL-NEXT: workitem_vgpr_count = 4 ; MOVREL-NEXT: reserved_vgpr_first = 0 ; MOVREL-NEXT: reserved_vgpr_count = 0 @@ -3168,21 +3168,24 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; MOVREL-NEXT: ; %bb.0: ; %entry ; MOVREL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; MOVREL-NEXT: s_load_dword s8, s[8:9], 0x8 +; MOVREL-NEXT: s_add_i32 s12, s12, s17 +; MOVREL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; MOVREL-NEXT: s_mov_b32 s4, 0 ; MOVREL-NEXT: s_mov_b32 s5, 0x40080000 -; MOVREL-NEXT: s_mov_b32 s2, 0 -; MOVREL-NEXT: s_mov_b32 s3, 0x40140000 ; MOVREL-NEXT: s_waitcnt lgkmcnt(0) ; MOVREL-NEXT: s_cmp_eq_u32 s8, 1 ; MOVREL-NEXT: s_cselect_b64 s[6:7], 2.0, 1.0 ; MOVREL-NEXT: s_cmp_eq_u32 s8, 2 +; MOVREL-NEXT: s_mov_b32 s2, 0 ; MOVREL-NEXT: s_cselect_b64 s[4:5], s[4:5], s[6:7] ; MOVREL-NEXT: s_cmp_eq_u32 s8, 3 +; MOVREL-NEXT: s_mov_b32 s3, 0x40140000 ; MOVREL-NEXT: s_cselect_b64 s[4:5], 4.0, s[4:5] ; MOVREL-NEXT: s_cmp_eq_u32 s8, 4 ; MOVREL-NEXT: s_cselect_b64 s[2:3], s[2:3], s[4:5] ; MOVREL-NEXT: v_mov_b32_e32 v0, s2 ; MOVREL-NEXT: v_mov_b32_e32 v3, s1 +; MOVREL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; MOVREL-NEXT: v_mov_b32_e32 v1, s3 ; MOVREL-NEXT: v_mov_b32_e32 v2, s0 ; MOVREL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -3210,7 +3213,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; GFX10-NEXT: enable_mem_ordered = 1 ; GFX10-NEXT: enable_fwd_progress = 0 ; GFX10-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; GFX10-NEXT: user_sgpr_count = 12 +; GFX10-NEXT: user_sgpr_count = 14 ; GFX10-NEXT: enable_trap_handler = 0 ; GFX10-NEXT: enable_sgpr_workgroup_id_x = 1 ; GFX10-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -3225,7 +3228,7 @@ define amdgpu_kernel void @dyn_extract_v5f64_s_s(ptr addrspace(1) %out, i32 %sel ; GFX10-NEXT: enable_sgpr_queue_ptr = 1 ; GFX10-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; GFX10-NEXT: enable_sgpr_dispatch_id = 1 -; GFX10-NEXT: enable_sgpr_flat_scratch_init = 0 +; GFX10-NEXT: enable_sgpr_flat_scratch_init = 1 ; GFX10-NEXT: enable_sgpr_private_segment_size = 0 ; GFX10-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; GFX10-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -4042,7 +4045,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; GPRIDX-NEXT: enable_mem_ordered = 0 ; GPRIDX-NEXT: enable_fwd_progress = 0 ; GPRIDX-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; GPRIDX-NEXT: user_sgpr_count = 12 +; GPRIDX-NEXT: user_sgpr_count = 14 ; GPRIDX-NEXT: enable_trap_handler = 0 ; GPRIDX-NEXT: enable_sgpr_workgroup_id_x = 1 ; GPRIDX-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -4057,7 +4060,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; GPRIDX-NEXT: enable_sgpr_queue_ptr = 1 ; GPRIDX-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; GPRIDX-NEXT: enable_sgpr_dispatch_id = 1 -; GPRIDX-NEXT: enable_sgpr_flat_scratch_init = 0 +; GPRIDX-NEXT: enable_sgpr_flat_scratch_init = 1 ; GPRIDX-NEXT: enable_sgpr_private_segment_size = 0 ; GPRIDX-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; GPRIDX-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -4074,7 +4077,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; GPRIDX-NEXT: gds_segment_byte_size = 0 ; GPRIDX-NEXT: kernarg_segment_byte_size = 28 ; GPRIDX-NEXT: workgroup_fbarrier_count = 0 -; GPRIDX-NEXT: wavefront_sgpr_count = 14 +; GPRIDX-NEXT: wavefront_sgpr_count = 16 ; GPRIDX-NEXT: workitem_vgpr_count = 2 ; GPRIDX-NEXT: reserved_vgpr_first = 0 ; GPRIDX-NEXT: reserved_vgpr_count = 0 @@ -4115,7 +4118,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: kernel_code_entry_byte_offset = 256 ; MOVREL-NEXT: kernel_code_prefetch_byte_size = 0 ; MOVREL-NEXT: granulated_workitem_vgpr_count = 0 -; MOVREL-NEXT: granulated_wavefront_sgpr_count = 1 +; MOVREL-NEXT: granulated_wavefront_sgpr_count = 2 ; MOVREL-NEXT: priority = 0 ; MOVREL-NEXT: float_mode = 240 ; MOVREL-NEXT: priv = 0 @@ -4126,7 +4129,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: enable_mem_ordered = 0 ; MOVREL-NEXT: enable_fwd_progress = 0 ; MOVREL-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; MOVREL-NEXT: user_sgpr_count = 12 +; MOVREL-NEXT: user_sgpr_count = 14 ; MOVREL-NEXT: enable_trap_handler = 0 ; MOVREL-NEXT: enable_sgpr_workgroup_id_x = 1 ; MOVREL-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -4141,7 +4144,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: enable_sgpr_queue_ptr = 1 ; MOVREL-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; MOVREL-NEXT: enable_sgpr_dispatch_id = 1 -; MOVREL-NEXT: enable_sgpr_flat_scratch_init = 0 +; MOVREL-NEXT: enable_sgpr_flat_scratch_init = 1 ; MOVREL-NEXT: enable_sgpr_private_segment_size = 0 ; MOVREL-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; MOVREL-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -4158,7 +4161,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: gds_segment_byte_size = 0 ; MOVREL-NEXT: kernarg_segment_byte_size = 28 ; MOVREL-NEXT: workgroup_fbarrier_count = 0 -; MOVREL-NEXT: wavefront_sgpr_count = 10 +; MOVREL-NEXT: wavefront_sgpr_count = 24 ; MOVREL-NEXT: workitem_vgpr_count = 3 ; MOVREL-NEXT: reserved_vgpr_first = 0 ; MOVREL-NEXT: reserved_vgpr_count = 0 @@ -4176,6 +4179,9 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: ; %bb.0: ; %entry ; MOVREL-NEXT: s_load_dword s2, s[8:9], 0x8 ; MOVREL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; MOVREL-NEXT: s_add_i32 s12, s12, s17 +; MOVREL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; MOVREL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; MOVREL-NEXT: s_waitcnt lgkmcnt(0) ; MOVREL-NEXT: s_cmp_eq_u32 s2, 1 ; MOVREL-NEXT: s_cselect_b32 s3, 2.0, 1.0 @@ -4211,7 +4217,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; GFX10-NEXT: enable_mem_ordered = 1 ; GFX10-NEXT: enable_fwd_progress = 0 ; GFX10-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; GFX10-NEXT: user_sgpr_count = 12 +; GFX10-NEXT: user_sgpr_count = 14 ; GFX10-NEXT: enable_trap_handler = 0 ; GFX10-NEXT: enable_sgpr_workgroup_id_x = 1 ; GFX10-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -4226,7 +4232,7 @@ define amdgpu_kernel void @dyn_extract_v4f32_s_s_s(ptr addrspace(1) %out, i32 %s ; GFX10-NEXT: enable_sgpr_queue_ptr = 1 ; GFX10-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; GFX10-NEXT: enable_sgpr_dispatch_id = 1 -; GFX10-NEXT: enable_sgpr_flat_scratch_init = 0 +; GFX10-NEXT: enable_sgpr_flat_scratch_init = 1 ; GFX10-NEXT: enable_sgpr_private_segment_size = 0 ; GFX10-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; GFX10-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -4387,7 +4393,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; GPRIDX-NEXT: enable_mem_ordered = 0 ; GPRIDX-NEXT: enable_fwd_progress = 0 ; GPRIDX-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; GPRIDX-NEXT: user_sgpr_count = 12 +; GPRIDX-NEXT: user_sgpr_count = 14 ; GPRIDX-NEXT: enable_trap_handler = 0 ; GPRIDX-NEXT: enable_sgpr_workgroup_id_x = 1 ; GPRIDX-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -4402,7 +4408,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; GPRIDX-NEXT: enable_sgpr_queue_ptr = 1 ; GPRIDX-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; GPRIDX-NEXT: enable_sgpr_dispatch_id = 1 -; GPRIDX-NEXT: enable_sgpr_flat_scratch_init = 0 +; GPRIDX-NEXT: enable_sgpr_flat_scratch_init = 1 ; GPRIDX-NEXT: enable_sgpr_private_segment_size = 0 ; GPRIDX-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; GPRIDX-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -4419,7 +4425,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; GPRIDX-NEXT: gds_segment_byte_size = 0 ; GPRIDX-NEXT: kernarg_segment_byte_size = 28 ; GPRIDX-NEXT: workgroup_fbarrier_count = 0 -; GPRIDX-NEXT: wavefront_sgpr_count = 14 +; GPRIDX-NEXT: wavefront_sgpr_count = 16 ; GPRIDX-NEXT: workitem_vgpr_count = 3 ; GPRIDX-NEXT: reserved_vgpr_first = 0 ; GPRIDX-NEXT: reserved_vgpr_count = 0 @@ -4463,7 +4469,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: kernel_code_entry_byte_offset = 256 ; MOVREL-NEXT: kernel_code_prefetch_byte_size = 0 ; MOVREL-NEXT: granulated_workitem_vgpr_count = 0 -; MOVREL-NEXT: granulated_wavefront_sgpr_count = 1 +; MOVREL-NEXT: granulated_wavefront_sgpr_count = 2 ; MOVREL-NEXT: priority = 0 ; MOVREL-NEXT: float_mode = 240 ; MOVREL-NEXT: priv = 0 @@ -4474,7 +4480,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: enable_mem_ordered = 0 ; MOVREL-NEXT: enable_fwd_progress = 0 ; MOVREL-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; MOVREL-NEXT: user_sgpr_count = 12 +; MOVREL-NEXT: user_sgpr_count = 14 ; MOVREL-NEXT: enable_trap_handler = 0 ; MOVREL-NEXT: enable_sgpr_workgroup_id_x = 1 ; MOVREL-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -4489,7 +4495,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: enable_sgpr_queue_ptr = 1 ; MOVREL-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; MOVREL-NEXT: enable_sgpr_dispatch_id = 1 -; MOVREL-NEXT: enable_sgpr_flat_scratch_init = 0 +; MOVREL-NEXT: enable_sgpr_flat_scratch_init = 1 ; MOVREL-NEXT: enable_sgpr_private_segment_size = 0 ; MOVREL-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; MOVREL-NEXT: enable_sgpr_grid_workgroup_count_y = 0 @@ -4506,7 +4512,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: gds_segment_byte_size = 0 ; MOVREL-NEXT: kernarg_segment_byte_size = 28 ; MOVREL-NEXT: workgroup_fbarrier_count = 0 -; MOVREL-NEXT: wavefront_sgpr_count = 10 +; MOVREL-NEXT: wavefront_sgpr_count = 24 ; MOVREL-NEXT: workitem_vgpr_count = 4 ; MOVREL-NEXT: reserved_vgpr_first = 0 ; MOVREL-NEXT: reserved_vgpr_count = 0 @@ -4524,10 +4530,12 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: ; %bb.0: ; %entry ; MOVREL-NEXT: s_load_dword s6, s[8:9], 0x8 ; MOVREL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; MOVREL-NEXT: s_add_i32 s12, s12, s17 +; MOVREL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; MOVREL-NEXT: s_mov_b32 s2, 0 -; MOVREL-NEXT: s_mov_b32 s3, 0x40080000 ; MOVREL-NEXT: s_waitcnt lgkmcnt(0) ; MOVREL-NEXT: s_cmp_eq_u32 s6, 1 +; MOVREL-NEXT: s_mov_b32 s3, 0x40080000 ; MOVREL-NEXT: s_cselect_b64 s[4:5], 2.0, 1.0 ; MOVREL-NEXT: s_cmp_eq_u32 s6, 2 ; MOVREL-NEXT: s_cselect_b64 s[2:3], s[2:3], s[4:5] @@ -4535,6 +4543,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; MOVREL-NEXT: s_cselect_b64 s[2:3], 4.0, s[2:3] ; MOVREL-NEXT: v_mov_b32_e32 v0, s2 ; MOVREL-NEXT: v_mov_b32_e32 v3, s1 +; MOVREL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; MOVREL-NEXT: v_mov_b32_e32 v1, s3 ; MOVREL-NEXT: v_mov_b32_e32 v2, s0 ; MOVREL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -4562,7 +4571,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; GFX10-NEXT: enable_mem_ordered = 1 ; GFX10-NEXT: enable_fwd_progress = 0 ; GFX10-NEXT: enable_sgpr_private_segment_wave_byte_offset = 0 -; GFX10-NEXT: user_sgpr_count = 12 +; GFX10-NEXT: user_sgpr_count = 14 ; GFX10-NEXT: enable_trap_handler = 0 ; GFX10-NEXT: enable_sgpr_workgroup_id_x = 1 ; GFX10-NEXT: enable_sgpr_workgroup_id_y = 1 @@ -4577,7 +4586,7 @@ define amdgpu_kernel void @dyn_extract_v4f64_s_s_s(ptr addrspace(1) %out, i32 %s ; GFX10-NEXT: enable_sgpr_queue_ptr = 1 ; GFX10-NEXT: enable_sgpr_kernarg_segment_ptr = 1 ; GFX10-NEXT: enable_sgpr_dispatch_id = 1 -; GFX10-NEXT: enable_sgpr_flat_scratch_init = 0 +; GFX10-NEXT: enable_sgpr_flat_scratch_init = 1 ; GFX10-NEXT: enable_sgpr_private_segment_size = 0 ; GFX10-NEXT: enable_sgpr_grid_workgroup_count_x = 0 ; GFX10-NEXT: enable_sgpr_grid_workgroup_count_y = 0 diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch-init.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch-init.ll index 00c44c27257bb..e207d95287783 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch-init.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/flat-scratch-init.ll @@ -35,7 +35,7 @@ define amdgpu_kernel void @stack_object_addrspacecast_in_kernel_no_calls() { ; RO-FLAT: scratch_store_dword ; RW-FLAT: .amdhsa_user_sgpr_private_segment_buffer 1 ; RO-FLAT-NOT: .amdhsa_user_sgpr_private_segment_buffer -; RW-FLAT: .amdhsa_user_sgpr_flat_scratch_init 1 +; RW-FLAT: .amdhsa_user_sgpr_flat_scratch_init 0 ; RO-FLAT-NOT: .amdhsa_user_sgpr_flat_scratch_init ; RW-FLAT: .amdhsa_system_sgpr_private_segment_wavefront_offset 1 ; RW-FLAT-NOT: .amdhsa_enable_private_segment @@ -43,7 +43,7 @@ define amdgpu_kernel void @stack_object_addrspacecast_in_kernel_no_calls() { ; RO-FLAT: .amdhsa_enable_private_segment 1 ; RW-FLAT: .amdhsa_reserve_flat_scratch 0 ; GCN: COMPUTE_PGM_RSRC2:SCRATCH_EN: 1 -; RW-FLAT: COMPUTE_PGM_RSRC2:USER_SGPR: 6 +; RW-FLAT: COMPUTE_PGM_RSRC2:USER_SGPR: 4 ; RO-FLAT: COMPUTE_PGM_RSRC2:USER_SGPR: 0 define amdgpu_kernel void @stack_object_in_kernel_no_calls() { %alloca = alloca i32, addrspace(5) diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/implicit-kernarg-backend-usage-global-isel.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/implicit-kernarg-backend-usage-global-isel.ll index 676035735d0af..86766e2904619 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/implicit-kernarg-backend-usage-global-isel.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/implicit-kernarg-backend-usage-global-isel.ll @@ -12,7 +12,9 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8V4-NEXT: s_load_dwordx2 s[2:3], s[6:7], 0x40 -; GFX8V4-NEXT: v_mov_b32_e32 v2, 1 +; GFX8V4-NEXT: s_add_i32 s12, s12, s17 +; GFX8V4-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX8V4-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX8V4-NEXT: s_waitcnt lgkmcnt(0) ; GFX8V4-NEXT: s_mov_b32 s4, s0 ; GFX8V4-NEXT: s_mov_b32 s5, s3 @@ -23,6 +25,7 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX8V4-NEXT: s_cmp_lg_u32 s1, -1 ; GFX8V4-NEXT: v_mov_b32_e32 v0, s4 ; GFX8V4-NEXT: s_cselect_b64 s[0:1], s[6:7], 0 +; GFX8V4-NEXT: v_mov_b32_e32 v2, 1 ; GFX8V4-NEXT: v_mov_b32_e32 v1, s5 ; GFX8V4-NEXT: flat_store_dword v[0:1], v2 ; GFX8V4-NEXT: s_waitcnt vmcnt(0) @@ -37,7 +40,9 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX8V5: ; %bb.0: ; GFX8V5-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8V5-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0xc8 -; GFX8V5-NEXT: v_mov_b32_e32 v2, 1 +; GFX8V5-NEXT: s_add_i32 s12, s12, s17 +; GFX8V5-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX8V5-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX8V5-NEXT: s_waitcnt lgkmcnt(0) ; GFX8V5-NEXT: s_mov_b32 s4, s0 ; GFX8V5-NEXT: s_mov_b32 s5, s2 @@ -47,6 +52,7 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX8V5-NEXT: s_cmp_lg_u32 s1, -1 ; GFX8V5-NEXT: v_mov_b32_e32 v0, s4 ; GFX8V5-NEXT: s_cselect_b64 s[0:1], s[2:3], 0 +; GFX8V5-NEXT: v_mov_b32_e32 v2, 1 ; GFX8V5-NEXT: v_mov_b32_e32 v1, s5 ; GFX8V5-NEXT: flat_store_dword v[0:1], v2 ; GFX8V5-NEXT: s_waitcnt vmcnt(0) @@ -60,9 +66,10 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX9V4-LABEL: addrspacecast: ; GFX9V4: ; %bb.0: ; GFX9V4-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9V4-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9V4-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9V4-NEXT: s_mov_b64 s[2:3], src_private_base ; GFX9V4-NEXT: s_mov_b64 s[4:5], src_shared_base -; GFX9V4-NEXT: v_mov_b32_e32 v2, 1 ; GFX9V4-NEXT: s_waitcnt lgkmcnt(0) ; GFX9V4-NEXT: s_mov_b32 s2, s0 ; GFX9V4-NEXT: s_cmp_lg_u32 s0, -1 @@ -71,6 +78,7 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX9V4-NEXT: s_cmp_lg_u32 s1, -1 ; GFX9V4-NEXT: v_mov_b32_e32 v0, s2 ; GFX9V4-NEXT: s_cselect_b64 s[0:1], s[4:5], 0 +; GFX9V4-NEXT: v_mov_b32_e32 v2, 1 ; GFX9V4-NEXT: v_mov_b32_e32 v1, s3 ; GFX9V4-NEXT: flat_store_dword v[0:1], v2 ; GFX9V4-NEXT: s_waitcnt vmcnt(0) @@ -84,9 +92,10 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX9V5-LABEL: addrspacecast: ; GFX9V5: ; %bb.0: ; GFX9V5-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9V5-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9V5-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9V5-NEXT: s_mov_b64 s[2:3], src_private_base ; GFX9V5-NEXT: s_mov_b64 s[4:5], src_shared_base -; GFX9V5-NEXT: v_mov_b32_e32 v2, 1 ; GFX9V5-NEXT: s_waitcnt lgkmcnt(0) ; GFX9V5-NEXT: s_mov_b32 s2, s0 ; GFX9V5-NEXT: s_cmp_lg_u32 s0, -1 @@ -95,6 +104,7 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ; GFX9V5-NEXT: s_cmp_lg_u32 s1, -1 ; GFX9V5-NEXT: v_mov_b32_e32 v0, s2 ; GFX9V5-NEXT: s_cselect_b64 s[0:1], s[4:5], 0 +; GFX9V5-NEXT: v_mov_b32_e32 v2, 1 ; GFX9V5-NEXT: v_mov_b32_e32 v1, s3 ; GFX9V5-NEXT: flat_store_dword v[0:1], v2 ; GFX9V5-NEXT: s_waitcnt vmcnt(0) @@ -111,7 +121,7 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ret void } -define amdgpu_kernel void @llvm_amdgcn_is_shared(ptr %ptr) { +define amdgpu_kernel void @llvm_amdgcn_is_shared(ptr %ptr) #0 { ; GFX8V4-LABEL: llvm_amdgcn_is_shared: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -167,7 +177,7 @@ define amdgpu_kernel void @llvm_amdgcn_is_shared(ptr %ptr) { ret void } -define amdgpu_kernel void @llvm_amdgcn_is_private(ptr %ptr) { +define amdgpu_kernel void @llvm_amdgcn_is_private(ptr %ptr) #0 { ; GFX8V4-LABEL: llvm_amdgcn_is_private: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -223,7 +233,7 @@ define amdgpu_kernel void @llvm_amdgcn_is_private(ptr %ptr) { ret void } -define amdgpu_kernel void @llvm_trap() { +define amdgpu_kernel void @llvm_trap() #0 { ; GFX8V4-LABEL: llvm_trap: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_mov_b64 s[0:1], s[6:7] @@ -246,7 +256,7 @@ define amdgpu_kernel void @llvm_trap() { unreachable } -define amdgpu_kernel void @llvm_debugtrap() { +define amdgpu_kernel void @llvm_debugtrap() #0 { ; GFX8V4-LABEL: llvm_debugtrap: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_trap 3 @@ -266,7 +276,7 @@ define amdgpu_kernel void @llvm_debugtrap() { unreachable } -define amdgpu_kernel void @llvm_amdgcn_queue_ptr(ptr addrspace(1) %ptr) { +define amdgpu_kernel void @llvm_amdgcn_queue_ptr(ptr addrspace(1) %ptr) #0 { ; GFX8V4-LABEL: llvm_amdgcn_queue_ptr: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: v_mov_b32_e32 v0, s6 @@ -374,3 +384,5 @@ declare void @llvm.debugtrap() !llvm.module.flags = !{!0} !0 = !{i32 1, !"amdhsa_code_object_version", i32 CODE_OBJECT_VERSION} + +attributes #0 = { "amdgpu-no-flat-scratch-init" } diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/insertelement-stack-lower.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/insertelement-stack-lower.ll index 378c6312c52be..94853767ccfac 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/insertelement-stack-lower.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/insertelement-stack-lower.ll @@ -9,7 +9,7 @@ define amdgpu_kernel void @v_insert_v64i32_varidx(ptr addrspace(1) %out.ptr, ptr ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[20:23], s[8:9], 0x0 ; GCN-NEXT: s_load_dwordx2 s[24:25], s[8:9], 0x10 -; GCN-NEXT: s_add_u32 s0, s0, s15 +; GCN-NEXT: s_add_u32 s0, s0, s17 ; GCN-NEXT: s_addc_u32 s1, s1, 0 ; GCN-NEXT: v_mov_b32_e32 v64, 0 ; GCN-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/lds-global-value.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/lds-global-value.ll index a6a7f35a774db..859f7ef16e395 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/lds-global-value.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/lds-global-value.ll @@ -11,13 +11,16 @@ define amdgpu_kernel void @use_lds_globals(ptr addrspace(1) %out, ptr addrspace( ; CHECK-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CHECK-NEXT: v_mov_b32_e32 v0, 4 ; CHECK-NEXT: s_mov_b32 m0, -1 +; CHECK-NEXT: s_add_i32 s12, s12, s17 ; CHECK-NEXT: ds_read_b32 v2, v0 -; CHECK-NEXT: v_mov_b32_e32 v3, 9 +; CHECK-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: s_add_u32 s0, s0, 4 ; CHECK-NEXT: s_addc_u32 s1, s1, 0 ; CHECK-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-NEXT: v_mov_b32_e32 v1, s1 +; CHECK-NEXT: v_mov_b32_e32 v3, 9 ; CHECK-NEXT: flat_store_dword v[0:1], v2 ; CHECK-NEXT: v_mov_b32_e32 v0, 0x200 ; CHECK-NEXT: ds_write_b32 v0, v3 diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.if.break.i64.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.if.break.i64.ll index dcc2c23cae046..a5a75f74833f1 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.if.break.i64.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.if.break.i64.ll @@ -6,6 +6,9 @@ define amdgpu_kernel void @test_wave64(i32 %arg0, [8 x i32], i64 %saved) { ; GCN: ; %bb.0: ; %entry ; GCN-NEXT: s_load_dword s2, s[8:9], 0x0 ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0xa +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_cmp_eq_u32 s2, 0 ; GCN-NEXT: s_cselect_b32 s2, 1, 0 diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.trig.preop.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.trig.preop.ll index ad588ebee2f9e..1deee215e522b 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.trig.preop.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.trig.preop.ll @@ -42,6 +42,9 @@ define amdgpu_kernel void @s_trig_preop_f64(double %a, i32 %b) { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_trig_preop_f64 v[0:1], s[0:1], v0 @@ -59,6 +62,9 @@ define amdgpu_kernel void @s_trig_preop_f64(double %a, i32 %b) { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_trig_preop_f64 v[0:1], s[0:1], v0 @@ -76,6 +82,8 @@ define amdgpu_kernel void @s_trig_preop_f64(double %a, i32 %b) { ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dword s2, s[8:9], 0x8 ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_mov_b32_e32 v0, s2 ; GFX9-NEXT: v_trig_preop_f64 v[0:1], s[0:1], v0 @@ -85,6 +93,10 @@ define amdgpu_kernel void @s_trig_preop_f64(double %a, i32 %b) { ; ; GFX10-LABEL: s_trig_preop_f64: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_clause 0x1 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: s_load_dword s2, s[8:9], 0x8 @@ -113,6 +125,9 @@ define amdgpu_kernel void @s_trig_preop_f64_imm(double %a, i32 %b) { ; CI-LABEL: s_trig_preop_f64_imm: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_trig_preop_f64 v[0:1], s[0:1], 7 ; CI-NEXT: s_add_u32 s0, s0, 4 @@ -128,6 +143,9 @@ define amdgpu_kernel void @s_trig_preop_f64_imm(double %a, i32 %b) { ; VI-LABEL: s_trig_preop_f64_imm: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_trig_preop_f64 v[0:1], s[0:1], 7 ; VI-NEXT: s_add_u32 s0, s0, 4 @@ -143,6 +161,8 @@ define amdgpu_kernel void @s_trig_preop_f64_imm(double %a, i32 %b) { ; GFX9-LABEL: s_trig_preop_f64_imm: ; GFX9: ; %bb.0: ; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX9-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: v_trig_preop_f64 v[0:1], s[0:1], 7 ; GFX9-NEXT: flat_store_dwordx2 v[0:1], v[0:1] @@ -151,6 +171,10 @@ define amdgpu_kernel void @s_trig_preop_f64_imm(double %a, i32 %b) { ; ; GFX10-LABEL: s_trig_preop_f64_imm: ; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s12, s12, s17 +; GFX10-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) ; GFX10-NEXT: v_trig_preop_f64 v[0:1], s[0:1], 7 diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/sdivrem.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/sdivrem.ll index 40f29c56c8f12..b59f85b2dfa38 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/sdivrem.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/sdivrem.ll @@ -7,6 +7,9 @@ define amdgpu_kernel void @sdivrem_i32(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8-LABEL: sdivrem_i32: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_ashr_i32 s6, s5, 31 ; GFX8-NEXT: s_add_i32 s0, s5, s6 @@ -146,6 +149,9 @@ define amdgpu_kernel void @sdivrem_i64(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8-LABEL: sdivrem_i64: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx8 s[4:11], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_ashr_i32 s2, s9, 31 ; GFX8-NEXT: s_ashr_i32 s12, s11, 31 @@ -617,6 +623,9 @@ define amdgpu_kernel void @sdivrem_v2i32(ptr addrspace(1) %out0, ptr addrspace(1 ; GFX8-LABEL: sdivrem_v2i32: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx8 s[4:11], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_ashr_i32 s2, s10, 31 ; GFX8-NEXT: s_add_i32 s0, s10, s2 @@ -845,6 +854,9 @@ define amdgpu_kernel void @sdivrem_v2i32(ptr addrspace(1) %out0, ptr addrspace(1 define amdgpu_kernel void @sdivrem_v4i32(ptr addrspace(1) %out0, ptr addrspace(1) %out1, <4 x i32> %x, <4 x i32> %y) { ; GFX8-LABEL: sdivrem_v4i32: ; GFX8: ; %bb.0: +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_load_dwordx8 s[12:19], s[8:9], 0x10 ; GFX8-NEXT: s_load_dwordx4 s[4:7], s[8:9], 0x0 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) @@ -1271,6 +1283,9 @@ define amdgpu_kernel void @sdivrem_v4i32(ptr addrspace(1) %out0, ptr addrspace(1 define amdgpu_kernel void @sdivrem_v2i64(ptr addrspace(1) %out0, ptr addrspace(1) %out1, <2 x i64> %x, <2 x i64> %y) { ; GFX8-LABEL: sdivrem_v2i64: ; GFX8: ; %bb.0: +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_load_dwordx8 s[12:19], s[8:9], 0x0 ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x20 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) @@ -2187,6 +2202,9 @@ define amdgpu_kernel void @sdiv_i8(ptr addrspace(1) %out0, ptr addrspace(1) %out ; GFX8-LABEL: sdiv_i8: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_i32 s0, s4, 0x80008 ; GFX8-NEXT: s_ashr_i32 s5, s0, 31 @@ -2332,6 +2350,9 @@ define amdgpu_kernel void @sdivrem_v2i8(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8-LABEL: sdivrem_v2i8: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s2, s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_i32 s0, s2, 0x80010 ; GFX8-NEXT: s_ashr_i32 s3, s0, 31 @@ -2596,6 +2617,9 @@ define amdgpu_kernel void @sdiv_i16(ptr addrspace(1) %out0, ptr addrspace(1) %ou ; GFX8-LABEL: sdiv_i16: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_i32 s0, s4, 0x100010 ; GFX8-NEXT: s_ashr_i32 s5, s0, 31 @@ -2741,6 +2765,9 @@ define amdgpu_kernel void @sdivrem_v2i16(ptr addrspace(1) %out0, ptr addrspace(1 ; GFX8-LABEL: sdivrem_v2i16: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_sext_i32_i16 s0, s3 ; GFX8-NEXT: s_ashr_i32 s10, s0, 31 @@ -3002,6 +3029,9 @@ define amdgpu_kernel void @sdivrem_i3(ptr addrspace(1) %out0, ptr addrspace(1) % ; GFX8-LABEL: sdivrem_i3: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_i32 s0, s4, 0x30008 ; GFX8-NEXT: s_ashr_i32 s5, s0, 31 @@ -3153,6 +3183,9 @@ define amdgpu_kernel void @sdivrem_i27(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8-LABEL: sdivrem_i27: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_i32 s0, s5, 0x1b0000 ; GFX8-NEXT: s_ashr_i32 s5, s0, 31 diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/udivrem.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/udivrem.ll index e3c1a52696b47..ff0114cfc3ddb 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/udivrem.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/udivrem.ll @@ -7,6 +7,9 @@ define amdgpu_kernel void @udivrem_i32(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8-LABEL: udivrem_i32: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_cvt_f32_u32_e32 v0, s5 ; GFX8-NEXT: s_sub_i32 s0, 0, s5 @@ -113,6 +116,9 @@ define amdgpu_kernel void @udivrem_i64(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8-LABEL: udivrem_i64: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx8 s[4:11], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_cvt_f32_u32_e32 v0, s11 ; GFX8-NEXT: v_cvt_f32_u32_e32 v1, s10 @@ -523,6 +529,9 @@ define amdgpu_kernel void @udivrem_v2i32(ptr addrspace(1) %out0, ptr addrspace(1 ; GFX8-LABEL: udivrem_v2i32: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx8 s[4:11], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_cvt_f32_u32_e32 v0, s10 ; GFX8-NEXT: v_cvt_f32_u32_e32 v1, s11 @@ -685,6 +694,9 @@ define amdgpu_kernel void @udivrem_v2i32(ptr addrspace(1) %out0, ptr addrspace(1 define amdgpu_kernel void @udivrem_v4i32(ptr addrspace(1) %out0, ptr addrspace(1) %out1, <4 x i32> %x, <4 x i32> %y) { ; GFX8-LABEL: udivrem_v4i32: ; GFX8: ; %bb.0: +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_load_dwordx8 s[12:19], s[8:9], 0x10 ; GFX8-NEXT: s_load_dwordx4 s[4:7], s[8:9], 0x0 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) @@ -980,6 +992,9 @@ define amdgpu_kernel void @udivrem_v4i32(ptr addrspace(1) %out0, ptr addrspace(1 define amdgpu_kernel void @udivrem_v2i64(ptr addrspace(1) %out0, ptr addrspace(1) %out1, <2 x i64> %x, <2 x i64> %y) { ; GFX8-LABEL: udivrem_v2i64: ; GFX8: ; %bb.0: +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_load_dwordx4 s[12:15], s[8:9], 0x20 ; GFX8-NEXT: s_load_dwordx8 s[4:11], s[8:9], 0x0 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) @@ -1772,6 +1787,9 @@ define amdgpu_kernel void @udiv_i8(ptr addrspace(1) %out0, ptr addrspace(1) %out ; GFX8-LABEL: udiv_i8: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_u32 s5, s4, 0x80008 ; GFX8-NEXT: v_cvt_f32_ubyte0_e32 v0, s5 @@ -1885,6 +1903,9 @@ define amdgpu_kernel void @udivrem_v2i8(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s0, s[8:9], 0x10 ; GFX8-NEXT: s_load_dwordx4 s[4:7], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_u32 s2, s0, 0x80010 ; GFX8-NEXT: v_cvt_f32_ubyte0_e32 v0, s2 @@ -2081,6 +2102,9 @@ define amdgpu_kernel void @udiv_i16(ptr addrspace(1) %out0, ptr addrspace(1) %ou ; GFX8-LABEL: udiv_i16: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_lshr_b32 s5, s4, 16 ; GFX8-NEXT: v_cvt_f32_u32_e32 v0, s5 @@ -2194,6 +2218,9 @@ define amdgpu_kernel void @udivrem_v2i16(ptr addrspace(1) %out0, ptr addrspace(1 ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x10 ; GFX8-NEXT: s_load_dwordx4 s[4:7], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_and_b32 s2, s1, 0xffff ; GFX8-NEXT: v_cvt_f32_u32_e32 v0, s2 @@ -2387,6 +2414,9 @@ define amdgpu_kernel void @udivrem_i3(ptr addrspace(1) %out0, ptr addrspace(1) % ; GFX8-LABEL: udivrem_i3: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_bfe_u32 s5, s4, 0x30008 ; GFX8-NEXT: v_cvt_f32_ubyte0_e32 v0, s5 @@ -2505,6 +2535,9 @@ define amdgpu_kernel void @udivrem_i27(ptr addrspace(1) %out0, ptr addrspace(1) ; GFX8-LABEL: udivrem_i27: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_and_b32 s5, s5, 0x7ffffff ; GFX8-NEXT: v_cvt_f32_u32_e32 v0, s5 diff --git a/llvm/test/CodeGen/AMDGPU/abi-attribute-hints-undefined-behavior.ll b/llvm/test/CodeGen/AMDGPU/abi-attribute-hints-undefined-behavior.ll index 29fb320bf1283..c78f0a4eb61e9 100644 --- a/llvm/test/CodeGen/AMDGPU/abi-attribute-hints-undefined-behavior.ll +++ b/llvm/test/CodeGen/AMDGPU/abi-attribute-hints-undefined-behavior.ll @@ -135,6 +135,9 @@ define amdgpu_kernel void @marked_kernel_use_workitem_id(ptr addrspace(1) %ptr) ; FIXEDABI-LABEL: marked_kernel_use_workitem_id: ; FIXEDABI: ; %bb.0: ; FIXEDABI-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x0 +; FIXEDABI-NEXT: s_add_i32 s6, s6, s11 +; FIXEDABI-NEXT: s_mov_b32 flat_scratch_lo, s7 +; FIXEDABI-NEXT: s_lshr_b32 flat_scratch_hi, s6, 8 ; FIXEDABI-NEXT: s_waitcnt lgkmcnt(0) ; FIXEDABI-NEXT: v_mov_b32_e32 v4, s1 ; FIXEDABI-NEXT: v_mov_b32_e32 v3, s0 @@ -181,16 +184,19 @@ define amdgpu_kernel void @marked_kernel_use_workgroup_id(ptr addrspace(1) %ptr) ; FIXEDABI-LABEL: marked_kernel_use_workgroup_id: ; FIXEDABI: ; %bb.0: ; FIXEDABI-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x0 -; FIXEDABI-NEXT: v_mov_b32_e32 v2, s6 +; FIXEDABI-NEXT: s_add_i32 s6, s6, s11 +; FIXEDABI-NEXT: s_mov_b32 flat_scratch_lo, s7 +; FIXEDABI-NEXT: s_lshr_b32 flat_scratch_hi, s6, 8 +; FIXEDABI-NEXT: v_mov_b32_e32 v2, s8 ; FIXEDABI-NEXT: s_waitcnt lgkmcnt(0) ; FIXEDABI-NEXT: v_mov_b32_e32 v0, s0 ; FIXEDABI-NEXT: v_mov_b32_e32 v1, s1 ; FIXEDABI-NEXT: flat_store_dword v[0:1], v2 ; FIXEDABI-NEXT: s_waitcnt vmcnt(0) -; FIXEDABI-NEXT: v_mov_b32_e32 v2, s7 +; FIXEDABI-NEXT: v_mov_b32_e32 v2, s9 ; FIXEDABI-NEXT: flat_store_dword v[0:1], v2 ; FIXEDABI-NEXT: s_waitcnt vmcnt(0) -; FIXEDABI-NEXT: v_mov_b32_e32 v2, s8 +; FIXEDABI-NEXT: v_mov_b32_e32 v2, s10 ; FIXEDABI-NEXT: flat_store_dword v[0:1], v2 ; FIXEDABI-NEXT: s_waitcnt vmcnt(0) ; FIXEDABI-NEXT: s_endpgm @@ -238,6 +244,9 @@ define void @marked_func_use_other_sgpr(ptr addrspace(1) %ptr) #0 { define amdgpu_kernel void @marked_kernel_use_other_sgpr(ptr addrspace(1) %ptr) #0 { ; FIXEDABI-LABEL: marked_kernel_use_other_sgpr: ; FIXEDABI: ; %bb.0: +; FIXEDABI-NEXT: s_add_i32 s6, s6, s11 +; FIXEDABI-NEXT: s_mov_b32 flat_scratch_lo, s7 +; FIXEDABI-NEXT: s_lshr_b32 flat_scratch_hi, s6, 8 ; FIXEDABI-NEXT: s_add_u32 s0, s4, 8 ; FIXEDABI-NEXT: flat_load_ubyte v0, v[0:1] glc ; FIXEDABI-NEXT: s_addc_u32 s1, s5, 0 @@ -261,7 +270,10 @@ define amdgpu_kernel void @marked_kernel_use_other_sgpr(ptr addrspace(1) %ptr) # define amdgpu_kernel void @marked_kernel_nokernargs_implicitarg_ptr() #0 { ; FIXEDABI-LABEL: marked_kernel_nokernargs_implicitarg_ptr: ; FIXEDABI: ; %bb.0: +; FIXEDABI-NEXT: s_add_i32 s4, s4, s9 ; FIXEDABI-NEXT: v_mov_b32_e32 v0, 0 +; FIXEDABI-NEXT: s_mov_b32 flat_scratch_lo, s5 +; FIXEDABI-NEXT: s_lshr_b32 flat_scratch_hi, s4, 8 ; FIXEDABI-NEXT: v_mov_b32_e32 v1, 0 ; FIXEDABI-NEXT: flat_load_ubyte v0, v[0:1] glc ; FIXEDABI-NEXT: s_endpgm diff --git a/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll b/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll index 59bd4e9ac8ce6..3eba47d7d7852 100644 --- a/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll +++ b/llvm/test/CodeGen/AMDGPU/addrspacecast-constantexpr.ll @@ -1,5 +1,4 @@ ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals -; RUN: opt -mtriple=amdgcn-unknown-amdhsa -S -amdgpu-annotate-kernel-features < %s | FileCheck -check-prefixes=HSA,AKF_HSA %s ; RUN: opt -mtriple=amdgcn-unknown-amdhsa -S -passes=amdgpu-attributor < %s | FileCheck -check-prefixes=HSA,ATTRIBUTOR_HSA %s declare void @llvm.memcpy.p1.p4.i32(ptr addrspace(1) nocapture, ptr addrspace(4) nocapture, i32, i1) #0 @@ -27,11 +26,6 @@ define amdgpu_kernel void @store_cast_0_flat_to_group_addrspacecast() #1 { } define amdgpu_kernel void @store_cast_0_group_to_flat_addrspacecast() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@store_cast_0_group_to_flat_addrspacecast -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: store i32 7, ptr addrspace(4) addrspacecast (ptr addrspace(3) null to ptr addrspace(4)), align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@store_cast_0_group_to_flat_addrspacecast ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: store i32 7, ptr addrspace(4) addrspacecast (ptr addrspace(3) null to ptr addrspace(4)), align 4 @@ -42,11 +36,6 @@ define amdgpu_kernel void @store_cast_0_group_to_flat_addrspacecast() #1 { } define amdgpu_kernel void @store_constant_cast_group_gv_to_flat() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@store_constant_cast_group_gv_to_flat -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: store i32 7, ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.i32 to ptr addrspace(4)), align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@store_constant_cast_group_gv_to_flat ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: store i32 7, ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.i32 to ptr addrspace(4)), align 4 @@ -57,11 +46,6 @@ define amdgpu_kernel void @store_constant_cast_group_gv_to_flat() #1 { } define amdgpu_kernel void @store_constant_cast_group_gv_gep_to_flat() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@store_constant_cast_group_gv_gep_to_flat -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: store i32 7, ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@store_constant_cast_group_gv_gep_to_flat ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: store i32 7, ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), align 4 @@ -92,12 +76,6 @@ define amdgpu_kernel void @store_constant_cast_global_gv_gep_to_flat() #1 { } define amdgpu_kernel void @load_constant_cast_group_gv_gep_to_flat(ptr addrspace(1) %out) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@load_constant_cast_group_gv_gep_to_flat -; AKF_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), align 4 -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[OUT]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@load_constant_cast_group_gv_gep_to_flat ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), align 4 @@ -110,12 +88,6 @@ define amdgpu_kernel void @load_constant_cast_group_gv_gep_to_flat(ptr addrspace } define amdgpu_kernel void @atomicrmw_constant_cast_group_gv_gep_to_flat(ptr addrspace(1) %out) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@atomicrmw_constant_cast_group_gv_gep_to_flat -; AKF_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = atomicrmw add ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), i32 1 seq_cst, align 4 -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[OUT]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@atomicrmw_constant_cast_group_gv_gep_to_flat ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = atomicrmw add ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), i32 1 seq_cst, align 4 @@ -128,13 +100,6 @@ define amdgpu_kernel void @atomicrmw_constant_cast_group_gv_gep_to_flat(ptr addr } define amdgpu_kernel void @cmpxchg_constant_cast_group_gv_gep_to_flat(ptr addrspace(1) %out) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@cmpxchg_constant_cast_group_gv_gep_to_flat -; AKF_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = cmpxchg ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), i32 0, i32 1 seq_cst seq_cst, align 4 -; AKF_HSA-NEXT: [[VAL0:%.*]] = extractvalue { i32, i1 } [[VAL]], 0 -; AKF_HSA-NEXT: store i32 [[VAL0]], ptr addrspace(1) [[OUT]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@cmpxchg_constant_cast_group_gv_gep_to_flat ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = cmpxchg ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), i32 0, i32 1 seq_cst seq_cst, align 4 @@ -149,11 +114,6 @@ define amdgpu_kernel void @cmpxchg_constant_cast_group_gv_gep_to_flat(ptr addrsp } define amdgpu_kernel void @memcpy_constant_cast_group_gv_gep_to_flat(ptr addrspace(1) %out) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@memcpy_constant_cast_group_gv_gep_to_flat -; AKF_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: call void @llvm.memcpy.p1.p4.i32(ptr addrspace(1) align 4 [[OUT]], ptr addrspace(4) align 4 getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), i32 32, i1 false) -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@memcpy_constant_cast_group_gv_gep_to_flat ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: call void @llvm.memcpy.p1.p4.i32(ptr addrspace(1) align 4 [[OUT]], ptr addrspace(4) align 4 getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), i32 32, i1 false) @@ -165,11 +125,6 @@ define amdgpu_kernel void @memcpy_constant_cast_group_gv_gep_to_flat(ptr addrspa ; Can't just search the pointer value define amdgpu_kernel void @store_value_constant_cast_lds_gv_gep_to_flat(ptr addrspace(1) %out) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@store_value_constant_cast_lds_gv_gep_to_flat -; AKF_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: store ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), ptr addrspace(1) [[OUT]], align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@store_value_constant_cast_lds_gv_gep_to_flat ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: store ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8), ptr addrspace(1) [[OUT]], align 8 @@ -181,11 +136,6 @@ define amdgpu_kernel void @store_value_constant_cast_lds_gv_gep_to_flat(ptr addr ; Can't just search pointer types define amdgpu_kernel void @store_ptrtoint_value_constant_cast_lds_gv_gep_to_flat(ptr addrspace(1) %out) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@store_ptrtoint_value_constant_cast_lds_gv_gep_to_flat -; AKF_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: store i64 ptrtoint (ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8) to i64), ptr addrspace(1) [[OUT]], align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@store_ptrtoint_value_constant_cast_lds_gv_gep_to_flat ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[OUT:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: store i64 ptrtoint (ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8) to i64), ptr addrspace(1) [[OUT]], align 8 @@ -197,11 +147,6 @@ define amdgpu_kernel void @store_ptrtoint_value_constant_cast_lds_gv_gep_to_flat ; Cast group to flat, do GEP, cast back to group define amdgpu_kernel void @store_constant_cast_group_gv_gep_to_flat_to_group() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@store_constant_cast_group_gv_gep_to_flat_to_group -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: store i32 7, ptr addrspace(3) addrspacecast (ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8) to ptr addrspace(3)), align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@store_constant_cast_group_gv_gep_to_flat_to_group ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: store i32 7, ptr addrspace(3) addrspacecast (ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8) to ptr addrspace(3)), align 4 @@ -212,10 +157,6 @@ define amdgpu_kernel void @store_constant_cast_group_gv_gep_to_flat_to_group() # } define ptr addrspace(3) @ret_constant_cast_group_gv_gep_to_flat_to_group() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@ret_constant_cast_group_gv_gep_to_flat_to_group -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: ret ptr addrspace(3) addrspacecast (ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8) to ptr addrspace(3)) -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@ret_constant_cast_group_gv_gep_to_flat_to_group ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: ret ptr addrspace(3) addrspacecast (ptr addrspace(4) getelementptr ([256 x i32], ptr addrspace(4) addrspacecast (ptr addrspace(3) @lds.arr to ptr addrspace(4)), i64 0, i64 8) to ptr addrspace(3)) @@ -229,14 +170,11 @@ attributes #1 = { nounwind } !llvm.module.flags = !{!0} !0 = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. -; AKF_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) } -; AKF_HSA: attributes #[[ATTR1]] = { nounwind } ;. ; ATTRIBUTOR_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) } ; ATTRIBUTOR_HSA: attributes #[[ATTR1]] = { nounwind "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } ; ATTRIBUTOR_HSA: attributes #[[ATTR2]] = { nounwind "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } ;. -; AKF_HSA: [[META0:![0-9]+]] = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. ; ATTRIBUTOR_HSA: [[META0:![0-9]+]] = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. diff --git a/llvm/test/CodeGen/AMDGPU/always-uniform.ll b/llvm/test/CodeGen/AMDGPU/always-uniform.ll index b6c0271e5f56f..4e7022710c671 100644 --- a/llvm/test/CodeGen/AMDGPU/always-uniform.ll +++ b/llvm/test/CodeGen/AMDGPU/always-uniform.ll @@ -8,8 +8,10 @@ define amdgpu_kernel void @readfirstlane_uniform(ptr addrspace(1) noalias nocapt ; GCN-LABEL: readfirstlane_uniform: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 ; GCN-NEXT: v_readfirstlane_b32 s4, v0 ; GCN-NEXT: s_mov_b32 s5, 0 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_lshl_b64 s[4:5], s[4:5], 2 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_add_u32 s0, s0, s4 @@ -18,6 +20,7 @@ define amdgpu_kernel void @readfirstlane_uniform(ptr addrspace(1) noalias nocapt ; GCN-NEXT: s_add_u32 s0, s2, 40 ; GCN-NEXT: s_addc_u32 s1, s3, 0 ; GCN-NEXT: v_mov_b32_e32 v0, s0 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: v_mov_b32_e32 v1, s1 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v2, s4 diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fold-binop-select.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fold-binop-select.ll index 7fdc012d4f1b5..e71bf15384727 100644 --- a/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fold-binop-select.ll +++ b/llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fold-binop-select.ll @@ -393,6 +393,9 @@ define amdgpu_kernel void @select_add_lhs_const_i16(i1 %cond) { ; GCN-LABEL: select_add_lhs_const_i16: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dword s0, s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_bitcmp1_b32 s0, 0 ; GCN-NEXT: s_movk_i32 s0, 0x80 diff --git a/llvm/test/CodeGen/AMDGPU/amdhsa-trap-num-sgprs.ll b/llvm/test/CodeGen/AMDGPU/amdhsa-trap-num-sgprs.ll index 3e19ee5567929..85b5c7c870b23 100644 --- a/llvm/test/CodeGen/AMDGPU/amdhsa-trap-num-sgprs.ll +++ b/llvm/test/CodeGen/AMDGPU/amdhsa-trap-num-sgprs.ll @@ -34,7 +34,7 @@ define amdgpu_kernel void @amdhsa_trap_num_sgprs( ptr addrspace(1) %out26, i32 %in26, ptr addrspace(1) %out27, i32 %in27, ptr addrspace(1) %out28, i32 %in28, - ptr addrspace(1) %out29, i32 %in29) { + ptr addrspace(1) %out29, i32 %in29) #0 { entry: store i32 %in0, ptr addrspace(1) %out0 store i32 %in1, ptr addrspace(1) %out1 @@ -68,3 +68,5 @@ entry: store i32 %in29, ptr addrspace(1) %out29 ret void } + +attributes #0 = { "amdgpu-no-flat-scratch-init" } diff --git a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll index 6d205921923d3..8389a8e86cb44 100644 --- a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll +++ b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa-call.ll @@ -1,5 +1,4 @@ ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals -; RUN: opt -mtriple=amdgcn-unknown-amdhsa -S -amdgpu-annotate-kernel-features < %s | FileCheck -check-prefixes=AKF_HSA %s ; RUN: opt -mtriple=amdgcn-unknown-amdhsa -S -passes=amdgpu-attributor < %s | FileCheck -check-prefixes=ATTRIBUTOR_HSA %s ; TODO: The test contains UB which is refined by the Attributor and should be removed. @@ -19,12 +18,6 @@ declare ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() #0 declare i64 @llvm.amdgcn.dispatch.id() #0 define void @use_workitem_id_x() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_workitem_id_x -; AKF_HSA-SAME: () #[[ATTR1:[0-9]+]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.x() -; AKF_HSA-NEXT: store volatile i32 [[VAL]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workitem_id_x ; ATTRIBUTOR_HSA-SAME: () #[[ATTR1:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.x() @@ -37,12 +30,6 @@ define void @use_workitem_id_x() #1 { } define void @use_workitem_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_workitem_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() -; AKF_HSA-NEXT: store volatile i32 [[VAL]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workitem_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() @@ -55,12 +42,6 @@ define void @use_workitem_id_y() #1 { } define void @use_workitem_id_z() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_workitem_id_z -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workitem_id_z ; ATTRIBUTOR_HSA-SAME: () #[[ATTR3:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.z() @@ -73,12 +54,6 @@ define void @use_workitem_id_z() #1 { } define void @use_workgroup_id_x() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_x -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() -; AKF_HSA-NEXT: store volatile i32 [[VAL]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_x ; ATTRIBUTOR_HSA-SAME: () #[[ATTR4:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() @@ -91,12 +66,6 @@ define void @use_workgroup_id_x() #1 { } define void @use_workgroup_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: store volatile i32 [[VAL]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR5:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() @@ -109,12 +78,6 @@ define void @use_workgroup_id_y() #1 { } define void @use_workgroup_id_z() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_z -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_z ; ATTRIBUTOR_HSA-SAME: () #[[ATTR6:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() @@ -127,12 +90,6 @@ define void @use_workgroup_id_z() #1 { } define void @use_dispatch_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_dispatch_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() -; AKF_HSA-NEXT: store volatile ptr addrspace(4) [[DISPATCH_PTR]], ptr addrspace(1) poison, align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_dispatch_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR7:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() @@ -145,12 +102,6 @@ define void @use_dispatch_ptr() #1 { } define void @use_queue_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_queue_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[QUEUE_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.queue.ptr() -; AKF_HSA-NEXT: store volatile ptr addrspace(4) [[QUEUE_PTR]], ptr addrspace(1) poison, align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_queue_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR8:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[QUEUE_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.queue.ptr() @@ -163,12 +114,6 @@ define void @use_queue_ptr() #1 { } define void @use_dispatch_id() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_dispatch_id -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i64 @llvm.amdgcn.dispatch.id() -; AKF_HSA-NEXT: store volatile i64 [[VAL]], ptr addrspace(1) poison, align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_dispatch_id ; ATTRIBUTOR_HSA-SAME: () #[[ATTR9:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i64 @llvm.amdgcn.dispatch.id() @@ -181,14 +126,6 @@ define void @use_dispatch_id() #1 { } define void @use_workgroup_id_y_workgroup_id_z() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_y_workgroup_id_z -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_workgroup_id_y_workgroup_id_z ; ATTRIBUTOR_HSA-SAME: () #[[ATTR10:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() @@ -205,11 +142,6 @@ define void @use_workgroup_id_y_workgroup_id_z() #1 { } define void @func_indirect_use_workitem_id_x() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workitem_id_x -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workitem_id_x() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workitem_id_x ; ATTRIBUTOR_HSA-SAME: () #[[ATTR1]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workitem_id_x() @@ -220,11 +152,6 @@ define void @func_indirect_use_workitem_id_x() #1 { } define void @kernel_indirect_use_workitem_id_x() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@kernel_indirect_use_workitem_id_x -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workitem_id_x() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kernel_indirect_use_workitem_id_x ; ATTRIBUTOR_HSA-SAME: () #[[ATTR1]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workitem_id_x() @@ -235,11 +162,6 @@ define void @kernel_indirect_use_workitem_id_x() #1 { } define void @func_indirect_use_workitem_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workitem_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workitem_id_y() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workitem_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workitem_id_y() @@ -250,11 +172,6 @@ define void @func_indirect_use_workitem_id_y() #1 { } define void @func_indirect_use_workitem_id_z() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workitem_id_z -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workitem_id_z() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workitem_id_z ; ATTRIBUTOR_HSA-SAME: () #[[ATTR3]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workitem_id_z() @@ -265,11 +182,6 @@ define void @func_indirect_use_workitem_id_z() #1 { } define void @func_indirect_use_workgroup_id_x() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_x -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workgroup_id_x() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_x ; ATTRIBUTOR_HSA-SAME: () #[[ATTR4]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workgroup_id_x() @@ -280,11 +192,6 @@ define void @func_indirect_use_workgroup_id_x() #1 { } define void @kernel_indirect_use_workgroup_id_x() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@kernel_indirect_use_workgroup_id_x -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workgroup_id_x() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kernel_indirect_use_workgroup_id_x ; ATTRIBUTOR_HSA-SAME: () #[[ATTR4]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workgroup_id_x() @@ -295,11 +202,6 @@ define void @kernel_indirect_use_workgroup_id_x() #1 { } define void @func_indirect_use_workgroup_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workgroup_id_y() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR5]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workgroup_id_y() @@ -310,11 +212,6 @@ define void @func_indirect_use_workgroup_id_y() #1 { } define void @func_indirect_use_workgroup_id_z() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_z -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_workgroup_id_z() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_z ; ATTRIBUTOR_HSA-SAME: () #[[ATTR6]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_workgroup_id_z() @@ -325,11 +222,6 @@ define void @func_indirect_use_workgroup_id_z() #1 { } define void @func_indirect_indirect_use_workgroup_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_indirect_use_workgroup_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @func_indirect_use_workgroup_id_y() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_indirect_use_workgroup_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR5]] { ; ATTRIBUTOR_HSA-NEXT: call void @func_indirect_use_workgroup_id_y() @@ -340,11 +232,6 @@ define void @func_indirect_indirect_use_workgroup_id_y() #1 { } define void @indirect_x2_use_workgroup_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@indirect_x2_use_workgroup_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @func_indirect_indirect_use_workgroup_id_y() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@indirect_x2_use_workgroup_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR5]] { ; ATTRIBUTOR_HSA-NEXT: call void @func_indirect_indirect_use_workgroup_id_y() @@ -355,11 +242,6 @@ define void @indirect_x2_use_workgroup_id_y() #1 { } define void @func_indirect_use_dispatch_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_dispatch_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_dispatch_ptr() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_dispatch_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR7]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_dispatch_ptr() @@ -370,11 +252,6 @@ define void @func_indirect_use_dispatch_ptr() #1 { } define void @func_indirect_use_queue_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_queue_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_queue_ptr() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_queue_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR8]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_queue_ptr() @@ -385,11 +262,6 @@ define void @func_indirect_use_queue_ptr() #1 { } define void @func_indirect_use_dispatch_id() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_dispatch_id -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_dispatch_id() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_dispatch_id ; ATTRIBUTOR_HSA-SAME: () #[[ATTR9]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_dispatch_id() @@ -400,11 +272,6 @@ define void @func_indirect_use_dispatch_id() #1 { } define void @func_indirect_use_workgroup_id_y_workgroup_id_z() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_y_workgroup_id_z -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @func_indirect_use_workgroup_id_y_workgroup_id_z() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_workgroup_id_y_workgroup_id_z ; ATTRIBUTOR_HSA-SAME: () #[[ATTR11:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: call void @func_indirect_use_workgroup_id_y_workgroup_id_z() @@ -415,13 +282,6 @@ define void @func_indirect_use_workgroup_id_y_workgroup_id_z() #1 { } define void @recursive_use_workitem_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@recursive_use_workitem_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() -; AKF_HSA-NEXT: store volatile i32 [[VAL]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: call void @recursive_use_workitem_id_y() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@recursive_use_workitem_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() @@ -436,11 +296,6 @@ define void @recursive_use_workitem_id_y() #1 { } define void @call_recursive_use_workitem_id_y() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@call_recursive_use_workitem_id_y -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @recursive_use_workitem_id_y() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@call_recursive_use_workitem_id_y ; ATTRIBUTOR_HSA-SAME: () #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: call void @recursive_use_workitem_id_y() @@ -451,12 +306,6 @@ define void @call_recursive_use_workitem_id_y() #1 { } define void @use_group_to_flat_addrspacecast(ptr addrspace(3) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast -; AKF_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr addrspace(4) -; AKF_HSA-NEXT: store volatile i32 0, ptr addrspace(4) [[STOF]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR12:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr addrspace(4) @@ -470,12 +319,6 @@ define void @use_group_to_flat_addrspacecast(ptr addrspace(3) %ptr) #1 { define void @use_group_to_flat_addrspacecast_gfx9(ptr addrspace(3) %ptr) #2 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast_gfx9 -; AKF_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR2:[0-9]+]] { -; AKF_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr addrspace(4) -; AKF_HSA-NEXT: store volatile i32 0, ptr addrspace(4) [[STOF]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast_gfx9 ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR13:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr addrspace(4) @@ -488,13 +331,6 @@ define void @use_group_to_flat_addrspacecast_gfx9(ptr addrspace(3) %ptr) #2 { } define void @use_group_to_flat_addrspacecast_queue_ptr_gfx9(ptr addrspace(3) %ptr) #2 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast_queue_ptr_gfx9 -; AKF_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR2]] { -; AKF_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr addrspace(4) -; AKF_HSA-NEXT: store volatile i32 0, ptr addrspace(4) [[STOF]], align 4 -; AKF_HSA-NEXT: call void @func_indirect_use_queue_ptr() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast_queue_ptr_gfx9 ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR14:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr addrspace(4) @@ -509,11 +345,6 @@ define void @use_group_to_flat_addrspacecast_queue_ptr_gfx9(ptr addrspace(3) %pt } define void @indirect_use_group_to_flat_addrspacecast() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@indirect_use_group_to_flat_addrspacecast -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_group_to_flat_addrspacecast(ptr addrspace(3) null) -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@indirect_use_group_to_flat_addrspacecast ; ATTRIBUTOR_HSA-SAME: () #[[ATTR12]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_group_to_flat_addrspacecast(ptr addrspace(3) null) @@ -524,11 +355,6 @@ define void @indirect_use_group_to_flat_addrspacecast() #1 { } define void @indirect_use_group_to_flat_addrspacecast_gfx9() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@indirect_use_group_to_flat_addrspacecast_gfx9 -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_group_to_flat_addrspacecast_gfx9(ptr addrspace(3) null) -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@indirect_use_group_to_flat_addrspacecast_gfx9 ; ATTRIBUTOR_HSA-SAME: () #[[ATTR11]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_group_to_flat_addrspacecast_gfx9(ptr addrspace(3) null) @@ -539,11 +365,6 @@ define void @indirect_use_group_to_flat_addrspacecast_gfx9() #1 { } define void @indirect_use_group_to_flat_addrspacecast_queue_ptr_gfx9() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@indirect_use_group_to_flat_addrspacecast_queue_ptr_gfx9 -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_group_to_flat_addrspacecast_queue_ptr_gfx9(ptr addrspace(3) null) -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@indirect_use_group_to_flat_addrspacecast_queue_ptr_gfx9 ; ATTRIBUTOR_HSA-SAME: () #[[ATTR8]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_group_to_flat_addrspacecast_queue_ptr_gfx9(ptr addrspace(3) null) @@ -554,12 +375,6 @@ define void @indirect_use_group_to_flat_addrspacecast_queue_ptr_gfx9() #1 { } define void @use_kernarg_segment_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_kernarg_segment_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[KERNARG_SEGMENT_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr() -; AKF_HSA-NEXT: store volatile ptr addrspace(4) [[KERNARG_SEGMENT_PTR]], ptr addrspace(1) poison, align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_kernarg_segment_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR11]] { ; ATTRIBUTOR_HSA-NEXT: [[KERNARG_SEGMENT_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.kernarg.segment.ptr() @@ -571,11 +386,6 @@ define void @use_kernarg_segment_ptr() #1 { ret void } define void @func_indirect_use_kernarg_segment_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_kernarg_segment_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_kernarg_segment_ptr() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_kernarg_segment_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR11]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_kernarg_segment_ptr() @@ -586,12 +396,6 @@ define void @func_indirect_use_kernarg_segment_ptr() #1 { } define amdgpu_kernel void @kern_use_implicitarg_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_use_implicitarg_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[IMPLICITARG_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() -; AKF_HSA-NEXT: store volatile ptr addrspace(4) [[IMPLICITARG_PTR]], ptr addrspace(1) poison, align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_use_implicitarg_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR12]] { ; ATTRIBUTOR_HSA-NEXT: [[IMPLICITARG_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() @@ -604,12 +408,6 @@ define amdgpu_kernel void @kern_use_implicitarg_ptr() #1 { } define void @use_implicitarg_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_implicitarg_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[IMPLICITARG_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() -; AKF_HSA-NEXT: store volatile ptr addrspace(4) [[IMPLICITARG_PTR]], ptr addrspace(1) poison, align 8 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_implicitarg_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR12]] { ; ATTRIBUTOR_HSA-NEXT: [[IMPLICITARG_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() @@ -622,11 +420,6 @@ define void @use_implicitarg_ptr() #1 { } define void @func_indirect_use_implicitarg_ptr() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_implicitarg_ptr -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: call void @use_implicitarg_ptr() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_implicitarg_ptr ; ATTRIBUTOR_HSA-SAME: () #[[ATTR12]] { ; ATTRIBUTOR_HSA-NEXT: call void @use_implicitarg_ptr() @@ -640,10 +433,6 @@ declare void @external.func() #3 ; This function gets deleted. define internal void @defined.func() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@defined.func -; AKF_HSA-SAME: () #[[ATTR3:[0-9]+]] { -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@defined.func ; ATTRIBUTOR_HSA-SAME: () #[[ATTR16:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: ret void @@ -652,11 +441,6 @@ define internal void @defined.func() #3 { } define void @func_call_external() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_call_external -; AKF_HSA-SAME: () #[[ATTR3]] { -; AKF_HSA-NEXT: call void @external.func() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_call_external ; ATTRIBUTOR_HSA-SAME: () #[[ATTR15:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: call void @external.func() @@ -667,11 +451,6 @@ define void @func_call_external() #3 { } define void @func_call_defined() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_call_defined -; AKF_HSA-SAME: () #[[ATTR3]] { -; AKF_HSA-NEXT: call void @defined.func() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_call_defined ; ATTRIBUTOR_HSA-SAME: () #[[ATTR16]] { ; ATTRIBUTOR_HSA-NEXT: call void @defined.func() @@ -681,11 +460,6 @@ define void @func_call_defined() #3 { ret void } define void @func_call_asm() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_call_asm -; AKF_HSA-SAME: () #[[ATTR3]] { -; AKF_HSA-NEXT: call void asm sideeffect "", ""() #[[ATTR3]] -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_call_asm ; ATTRIBUTOR_HSA-SAME: () #[[ATTR16]] { ; ATTRIBUTOR_HSA-NEXT: call void asm sideeffect "", ""() #[[ATTR26:[0-9]+]] @@ -696,11 +470,6 @@ define void @func_call_asm() #3 { } define amdgpu_kernel void @kern_call_external() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_call_external -; AKF_HSA-SAME: () #[[ATTR4:[0-9]+]] { -; AKF_HSA-NEXT: call void @external.func() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_call_external ; ATTRIBUTOR_HSA-SAME: () #[[ATTR15]] { ; ATTRIBUTOR_HSA-NEXT: call void @external.func() @@ -711,11 +480,6 @@ define amdgpu_kernel void @kern_call_external() #3 { } define amdgpu_kernel void @func_kern_defined() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_kern_defined -; AKF_HSA-SAME: () #[[ATTR4]] { -; AKF_HSA-NEXT: call void @defined.func() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_kern_defined ; ATTRIBUTOR_HSA-SAME: () #[[ATTR17:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: call void @defined.func() @@ -726,12 +490,6 @@ define amdgpu_kernel void @func_kern_defined() #3 { } define i32 @use_dispatch_ptr_ret_type() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_dispatch_ptr_ret_type -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() -; AKF_HSA-NEXT: store volatile ptr addrspace(4) [[DISPATCH_PTR]], ptr addrspace(1) poison, align 8 -; AKF_HSA-NEXT: ret i32 0 -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_dispatch_ptr_ret_type ; ATTRIBUTOR_HSA-SAME: () #[[ATTR7]] { ; ATTRIBUTOR_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() @@ -744,12 +502,6 @@ define i32 @use_dispatch_ptr_ret_type() #1 { } define float @func_indirect_use_dispatch_ptr_constexpr_cast_func() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_use_dispatch_ptr_constexpr_cast_func -; AKF_HSA-SAME: () #[[ATTR1]] { -; AKF_HSA-NEXT: [[F:%.*]] = call float @use_dispatch_ptr_ret_type() -; AKF_HSA-NEXT: [[FADD:%.*]] = fadd float [[F]], 1.000000e+00 -; AKF_HSA-NEXT: ret float [[FADD]] -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_use_dispatch_ptr_constexpr_cast_func ; ATTRIBUTOR_HSA-SAME: () #[[ATTR7]] { ; ATTRIBUTOR_HSA-NEXT: [[F:%.*]] = call float @use_dispatch_ptr_ret_type() @@ -762,12 +514,6 @@ define float @func_indirect_use_dispatch_ptr_constexpr_cast_func() #1 { } define float @func_indirect_call(ptr %fptr) #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_call -; AKF_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR3]] { -; AKF_HSA-NEXT: [[F:%.*]] = call float [[FPTR]]() -; AKF_HSA-NEXT: [[FADD:%.*]] = fadd float [[F]], 1.000000e+00 -; AKF_HSA-NEXT: ret float [[FADD]] -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_call ; ATTRIBUTOR_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR15]] { ; ATTRIBUTOR_HSA-NEXT: [[F:%.*]] = call float [[FPTR]]() @@ -781,12 +527,6 @@ define float @func_indirect_call(ptr %fptr) #3 { declare float @extern() #3 define float @func_extern_call() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_extern_call -; AKF_HSA-SAME: () #[[ATTR3]] { -; AKF_HSA-NEXT: [[F:%.*]] = call float @extern() -; AKF_HSA-NEXT: [[FADD:%.*]] = fadd float [[F]], 1.000000e+00 -; AKF_HSA-NEXT: ret float [[FADD]] -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_extern_call ; ATTRIBUTOR_HSA-SAME: () #[[ATTR15]] { ; ATTRIBUTOR_HSA-NEXT: [[F:%.*]] = call float @extern() @@ -799,12 +539,6 @@ define float @func_extern_call() #3 { } define float @func_null_call(ptr %fptr) #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_null_call -; AKF_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR3]] { -; AKF_HSA-NEXT: [[F:%.*]] = call float null() -; AKF_HSA-NEXT: [[FADD:%.*]] = fadd float [[F]], 1.000000e+00 -; AKF_HSA-NEXT: ret float [[FADD]] -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_null_call ; ATTRIBUTOR_HSA-SAME: (ptr [[FPTR:%.*]]) #[[ATTR15]] { ; ATTRIBUTOR_HSA-NEXT: [[F:%.*]] = call float null() @@ -820,12 +554,6 @@ declare float @llvm.amdgcn.rcp.f32(float) #0 ; Calls some other recognized intrinsic define float @func_other_intrinsic_call(float %arg) #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_other_intrinsic_call -; AKF_HSA-SAME: (float [[ARG:%.*]]) #[[ATTR3]] { -; AKF_HSA-NEXT: [[F:%.*]] = call float @llvm.amdgcn.rcp.f32(float [[ARG]]) -; AKF_HSA-NEXT: [[FADD:%.*]] = fadd float [[F]], 1.000000e+00 -; AKF_HSA-NEXT: ret float [[FADD]] -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_other_intrinsic_call ; ATTRIBUTOR_HSA-SAME: (float [[ARG:%.*]]) #[[ATTR16]] { ; ATTRIBUTOR_HSA-NEXT: [[F:%.*]] = call float @llvm.amdgcn.rcp.f32(float [[ARG]]) @@ -839,11 +567,6 @@ define float @func_other_intrinsic_call(float %arg) #3 { ; Hostcall needs to be enabled for sanitizers define amdgpu_kernel void @kern_sanitize_address() #4 { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_sanitize_address -; AKF_HSA-SAME: () #[[ATTR5:[0-9]+]] { -; AKF_HSA-NEXT: store volatile i32 0, ptr addrspace(1) null, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_sanitize_address ; ATTRIBUTOR_HSA-SAME: () #[[ATTR18:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: store volatile i32 0, ptr addrspace(1) null, align 4 @@ -855,11 +578,6 @@ define amdgpu_kernel void @kern_sanitize_address() #4 { ; Hostcall needs to be enabled for sanitizers define void @func_sanitize_address() #4 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_sanitize_address -; AKF_HSA-SAME: () #[[ATTR5]] { -; AKF_HSA-NEXT: store volatile i32 0, ptr addrspace(1) null, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_sanitize_address ; ATTRIBUTOR_HSA-SAME: () #[[ATTR18]] { ; ATTRIBUTOR_HSA-NEXT: store volatile i32 0, ptr addrspace(1) null, align 4 @@ -871,11 +589,6 @@ define void @func_sanitize_address() #4 { ; Hostcall needs to be enabled for sanitizers define void @func_indirect_sanitize_address() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@func_indirect_sanitize_address -; AKF_HSA-SAME: () #[[ATTR3]] { -; AKF_HSA-NEXT: call void @func_sanitize_address() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@func_indirect_sanitize_address ; ATTRIBUTOR_HSA-SAME: () #[[ATTR19:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: call void @func_sanitize_address() @@ -887,11 +600,6 @@ define void @func_indirect_sanitize_address() #3 { ; Hostcall needs to be enabled for sanitizers define amdgpu_kernel void @kern_indirect_sanitize_address() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_indirect_sanitize_address -; AKF_HSA-SAME: () #[[ATTR4]] { -; AKF_HSA-NEXT: call void @func_sanitize_address() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_indirect_sanitize_address ; ATTRIBUTOR_HSA-SAME: () #[[ATTR19]] { ; ATTRIBUTOR_HSA-NEXT: call void @func_sanitize_address() @@ -906,11 +614,6 @@ define amdgpu_kernel void @kern_indirect_sanitize_address() #3 { declare void @extern_func_sanitize_address() #5 define amdgpu_kernel void @kern_decl_sanitize_address() #3 { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_decl_sanitize_address -; AKF_HSA-SAME: () #[[ATTR4]] { -; AKF_HSA-NEXT: call void @extern_func_sanitize_address() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_decl_sanitize_address ; ATTRIBUTOR_HSA-SAME: () #[[ATTR15]] { ; ATTRIBUTOR_HSA-NEXT: call void @extern_func_sanitize_address() @@ -923,10 +626,6 @@ define amdgpu_kernel void @kern_decl_sanitize_address() #3 { declare void @enqueue_block_decl() #6 define internal void @enqueue_block_def() #6 { -; AKF_HSA-LABEL: define {{[^@]+}}@enqueue_block_def -; AKF_HSA-SAME: () #[[ATTR7:[0-9]+]] { -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@enqueue_block_def ; ATTRIBUTOR_HSA-SAME: () #[[ATTR22:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: ret void @@ -935,11 +634,6 @@ define internal void @enqueue_block_def() #6 { } define amdgpu_kernel void @kern_call_enqueued_block_decl() { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_call_enqueued_block_decl -; AKF_HSA-SAME: () #[[ATTR8:[0-9]+]] { -; AKF_HSA-NEXT: call void @enqueue_block_decl() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_call_enqueued_block_decl ; ATTRIBUTOR_HSA-SAME: () #[[ATTR23:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: call void @enqueue_block_decl() @@ -950,11 +644,6 @@ define amdgpu_kernel void @kern_call_enqueued_block_decl() { } define amdgpu_kernel void @kern_call_enqueued_block_def() { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_call_enqueued_block_def -; AKF_HSA-SAME: () #[[ATTR8]] { -; AKF_HSA-NEXT: call void @enqueue_block_def() -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_call_enqueued_block_def ; ATTRIBUTOR_HSA-SAME: () #[[ATTR24:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: call void @enqueue_block_def() @@ -965,9 +654,6 @@ define amdgpu_kernel void @kern_call_enqueued_block_def() { } define void @unused_enqueue_block() { -; AKF_HSA-LABEL: define {{[^@]+}}@unused_enqueue_block() { -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@unused_enqueue_block ; ATTRIBUTOR_HSA-SAME: () #[[ATTR25:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: ret void @@ -976,9 +662,6 @@ define void @unused_enqueue_block() { } define internal void @known_func() { -; AKF_HSA-LABEL: define {{[^@]+}}@known_func() { -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@known_func ; ATTRIBUTOR_HSA-SAME: () #[[ATTR25]] { ; ATTRIBUTOR_HSA-NEXT: ret void @@ -988,11 +671,6 @@ define internal void @known_func() { ; Should never happen define amdgpu_kernel void @kern_callsite_enqueue_block() { -; AKF_HSA-LABEL: define {{[^@]+}}@kern_callsite_enqueue_block -; AKF_HSA-SAME: () #[[ATTR8]] { -; AKF_HSA-NEXT: call void @known_func() #[[ATTR7]] -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@kern_callsite_enqueue_block ; ATTRIBUTOR_HSA-SAME: () #[[ATTR24]] { ; ATTRIBUTOR_HSA-NEXT: call void @known_func() #[[ATTR27:[0-9]+]] @@ -1014,15 +692,6 @@ attributes #6 = { "enqueued-block" } !llvm.module.flags = !{!0} !0 = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. -; AKF_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } -; AKF_HSA: attributes #[[ATTR1]] = { nounwind "target-cpu"="fiji" } -; AKF_HSA: attributes #[[ATTR2]] = { nounwind "target-cpu"="gfx900" } -; AKF_HSA: attributes #[[ATTR3]] = { nounwind } -; AKF_HSA: attributes #[[ATTR4]] = { nounwind "amdgpu-calls" } -; AKF_HSA: attributes #[[ATTR5]] = { nounwind sanitize_address } -; AKF_HSA: attributes #[[ATTR6:[0-9]+]] = { nounwind sanitize_address "amdgpu-no-implicitarg-ptr" } -; AKF_HSA: attributes #[[ATTR7]] = { "enqueued-block" } -; AKF_HSA: attributes #[[ATTR8]] = { "amdgpu-calls" } ;. ; ATTRIBUTOR_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } ; ATTRIBUTOR_HSA: attributes #[[ATTR1]] = { nounwind "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "target-cpu"="fiji" "uniform-work-group-size"="false" } diff --git a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa.ll b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa.ll index 2809f0957462a..32bb22b699b61 100644 --- a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa.ll +++ b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features-hsa.ll @@ -1,5 +1,4 @@ ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals -; RUN: opt -mtriple=amdgcn-unknown-amdhsa -S -amdgpu-annotate-kernel-features < %s | FileCheck -check-prefixes=HSA,AKF_HSA %s ; RUN: opt -mtriple=amdgcn-unknown-amdhsa -S -passes=amdgpu-attributor < %s | FileCheck -check-prefixes=HSA,ATTRIBUTOR_HSA %s target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5" @@ -33,12 +32,6 @@ define amdgpu_kernel void @use_tgid_x(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_y(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tgid_y -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tgid_y ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR2:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() @@ -51,14 +44,6 @@ define amdgpu_kernel void @use_tgid_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @multi_use_tgid_y(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@multi_use_tgid_y -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@multi_use_tgid_y ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() @@ -75,14 +60,6 @@ define amdgpu_kernel void @multi_use_tgid_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_x_y(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tgid_x_y -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tgid_x_y ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() @@ -99,12 +76,6 @@ define amdgpu_kernel void @use_tgid_x_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_z(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tgid_z -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tgid_z ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR3:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() @@ -117,14 +88,6 @@ define amdgpu_kernel void @use_tgid_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_x_z(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tgid_x_z -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tgid_x_z ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR3]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() @@ -141,14 +104,6 @@ define amdgpu_kernel void @use_tgid_x_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_y_z(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tgid_y_z -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tgid_y_z ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR4:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() @@ -165,16 +120,6 @@ define amdgpu_kernel void @use_tgid_y_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_x_y_z(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tgid_x_y_z -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: [[VAL2:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL2]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tgid_x_y_z ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR4]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() @@ -207,12 +152,6 @@ define amdgpu_kernel void @use_tidig_x(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_y(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tidig_y -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tidig_y ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR5:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() @@ -225,12 +164,6 @@ define amdgpu_kernel void @use_tidig_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_z(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tidig_z -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.z() -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tidig_z ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR6:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL:%.*]] = call i32 @llvm.amdgcn.workitem.id.z() @@ -259,14 +192,6 @@ define amdgpu_kernel void @use_tidig_x_tgid_x(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_y_tgid_y(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tidig_y_tgid_y -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tidig_y_tgid_y ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR7:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() @@ -283,16 +208,6 @@ define amdgpu_kernel void @use_tidig_y_tgid_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_x_y_z(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_tidig_x_y_z -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workitem.id.x() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() -; AKF_HSA-NEXT: [[VAL2:%.*]] = call i32 @llvm.amdgcn.workitem.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL2]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_tidig_x_y_z ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR8:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workitem.id.x() @@ -313,22 +228,6 @@ define amdgpu_kernel void @use_tidig_x_y_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_all_workitems(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_all_workitems -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workitem.id.x() -; AKF_HSA-NEXT: [[VAL1:%.*]] = call i32 @llvm.amdgcn.workitem.id.y() -; AKF_HSA-NEXT: [[VAL2:%.*]] = call i32 @llvm.amdgcn.workitem.id.z() -; AKF_HSA-NEXT: [[VAL3:%.*]] = call i32 @llvm.amdgcn.workgroup.id.x() -; AKF_HSA-NEXT: [[VAL4:%.*]] = call i32 @llvm.amdgcn.workgroup.id.y() -; AKF_HSA-NEXT: [[VAL5:%.*]] = call i32 @llvm.amdgcn.workgroup.id.z() -; AKF_HSA-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL2]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL3]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL4]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: store volatile i32 [[VAL5]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_all_workitems ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR9:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[VAL0:%.*]] = call i32 @llvm.amdgcn.workitem.id.x() @@ -361,13 +260,6 @@ define amdgpu_kernel void @use_all_workitems(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_dispatch_ptr(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_dispatch_ptr -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() -; AKF_HSA-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(4) [[DISPATCH_PTR]], align 4 -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_dispatch_ptr ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR10:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() @@ -382,13 +274,6 @@ define amdgpu_kernel void @use_dispatch_ptr(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_queue_ptr(ptr addrspace(1) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_queue_ptr -; AKF_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.queue.ptr() -; AKF_HSA-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(4) [[DISPATCH_PTR]], align 4 -; AKF_HSA-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_queue_ptr ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR11:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[DISPATCH_PTR:%.*]] = call ptr addrspace(4) @llvm.amdgcn.queue.ptr() @@ -417,12 +302,6 @@ define amdgpu_kernel void @use_kernarg_segment_ptr(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_group_to_flat_addrspacecast(ptr addrspace(3) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast -; AKF_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr -; AKF_HSA-NEXT: store volatile i32 0, ptr [[STOF]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_group_to_flat_addrspacecast ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(3) [[PTR:%.*]]) #[[ATTR12:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(3) [[PTR]] to ptr @@ -435,12 +314,6 @@ define amdgpu_kernel void @use_group_to_flat_addrspacecast(ptr addrspace(3) %ptr } define amdgpu_kernel void @use_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_private_to_flat_addrspacecast -; AKF_HSA-SAME: (ptr addrspace(5) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(5) [[PTR]] to ptr -; AKF_HSA-NEXT: store volatile i32 0, ptr [[STOF]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_private_to_flat_addrspacecast ; ATTRIBUTOR_HSA-SAME: (ptr addrspace(5) [[PTR:%.*]]) #[[ATTR13:[0-9]+]] { ; ATTRIBUTOR_HSA-NEXT: [[STOF:%.*]] = addrspacecast ptr addrspace(5) [[PTR]] to ptr @@ -526,13 +399,6 @@ define amdgpu_kernel void @use_flat_to_constant_addrspacecast(ptr %ptr) #1 { } define amdgpu_kernel void @use_is_shared(ptr %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_is_shared -; AKF_HSA-SAME: (ptr [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[IS_SHARED:%.*]] = call i1 @llvm.amdgcn.is.shared(ptr [[PTR]]) -; AKF_HSA-NEXT: [[EXT:%.*]] = zext i1 [[IS_SHARED]] to i32 -; AKF_HSA-NEXT: store i32 [[EXT]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_is_shared ; ATTRIBUTOR_HSA-SAME: (ptr [[PTR:%.*]]) #[[ATTR12]] { ; ATTRIBUTOR_HSA-NEXT: [[IS_SHARED:%.*]] = call i1 @llvm.amdgcn.is.shared(ptr [[PTR]]) @@ -547,13 +413,6 @@ define amdgpu_kernel void @use_is_shared(ptr %ptr) #1 { } define amdgpu_kernel void @use_is_private(ptr %ptr) #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_is_private -; AKF_HSA-SAME: (ptr [[PTR:%.*]]) #[[ATTR1]] { -; AKF_HSA-NEXT: [[IS_PRIVATE:%.*]] = call i1 @llvm.amdgcn.is.private(ptr [[PTR]]) -; AKF_HSA-NEXT: [[EXT:%.*]] = zext i1 [[IS_PRIVATE]] to i32 -; AKF_HSA-NEXT: store i32 [[EXT]], ptr addrspace(1) poison, align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_is_private ; ATTRIBUTOR_HSA-SAME: (ptr [[PTR:%.*]]) #[[ATTR12]] { ; ATTRIBUTOR_HSA-NEXT: [[IS_PRIVATE:%.*]] = call i1 @llvm.amdgcn.is.private(ptr [[PTR]]) @@ -568,12 +427,6 @@ define amdgpu_kernel void @use_is_private(ptr %ptr) #1 { } define amdgpu_kernel void @use_alloca() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_alloca -; AKF_HSA-SAME: () #[[ATTR2:[0-9]+]] { -; AKF_HSA-NEXT: [[ALLOCA:%.*]] = alloca i32, align 4, addrspace(5) -; AKF_HSA-NEXT: store i32 0, ptr addrspace(5) [[ALLOCA]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_alloca ; ATTRIBUTOR_HSA-SAME: () #[[ATTR1]] { ; ATTRIBUTOR_HSA-NEXT: [[ALLOCA:%.*]] = alloca i32, align 4, addrspace(5) @@ -586,15 +439,6 @@ define amdgpu_kernel void @use_alloca() #1 { } define amdgpu_kernel void @use_alloca_non_entry_block() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_alloca_non_entry_block -; AKF_HSA-SAME: () #[[ATTR2]] { -; AKF_HSA-NEXT: entry: -; AKF_HSA-NEXT: br label [[BB:%.*]] -; AKF_HSA: bb: -; AKF_HSA-NEXT: [[ALLOCA:%.*]] = alloca i32, align 4, addrspace(5) -; AKF_HSA-NEXT: store i32 0, ptr addrspace(5) [[ALLOCA]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_alloca_non_entry_block ; ATTRIBUTOR_HSA-SAME: () #[[ATTR1]] { ; ATTRIBUTOR_HSA-NEXT: entry: @@ -614,12 +458,6 @@ bb: } define void @use_alloca_func() #1 { -; AKF_HSA-LABEL: define {{[^@]+}}@use_alloca_func -; AKF_HSA-SAME: () #[[ATTR2]] { -; AKF_HSA-NEXT: [[ALLOCA:%.*]] = alloca i32, align 4, addrspace(5) -; AKF_HSA-NEXT: store i32 0, ptr addrspace(5) [[ALLOCA]], align 4 -; AKF_HSA-NEXT: ret void -; ; ATTRIBUTOR_HSA-LABEL: define {{[^@]+}}@use_alloca_func ; ATTRIBUTOR_HSA-SAME: () #[[ATTR1]] { ; ATTRIBUTOR_HSA-NEXT: [[ALLOCA:%.*]] = alloca i32, align 4, addrspace(5) @@ -638,9 +476,6 @@ attributes #1 = { nounwind } !0 = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. -; AKF_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } -; AKF_HSA: attributes #[[ATTR1]] = { nounwind } -; AKF_HSA: attributes #[[ATTR2]] = { nounwind "amdgpu-stack-objects" } ;. ; ATTRIBUTOR_HSA: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } ; ATTRIBUTOR_HSA: attributes #[[ATTR1]] = { nounwind "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } diff --git a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features.ll b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features.ll index 20ce05278d213..15dc1a0529254 100644 --- a/llvm/test/CodeGen/AMDGPU/annotate-kernel-features.ll +++ b/llvm/test/CodeGen/AMDGPU/annotate-kernel-features.ll @@ -1,5 +1,4 @@ ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals -; RUN: opt -S -mtriple=amdgcn-unknown-unknown -amdgpu-annotate-kernel-features < %s | FileCheck -check-prefixes=CHECK,AKF_CHECK %s ; RUN: opt -S -mtriple=amdgcn-unknown-unknown -passes=amdgpu-attributor < %s | FileCheck -check-prefixes=CHECK,ATTRIBUTOR_CHECK %s declare i32 @llvm.r600.read.tgid.x() #0 @@ -27,12 +26,6 @@ define amdgpu_kernel void @use_tgid_x(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_y(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tgid_y -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tgid_y ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR2:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tgid.y() @@ -45,14 +38,6 @@ define amdgpu_kernel void @use_tgid_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @multi_use_tgid_y(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@multi_use_tgid_y -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@multi_use_tgid_y ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.y() @@ -69,14 +54,6 @@ define amdgpu_kernel void @multi_use_tgid_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_x_y(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tgid_x_y -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.x() -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tgid_x_y ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR2]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.x() @@ -93,12 +70,6 @@ define amdgpu_kernel void @use_tgid_x_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_z(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tgid_z -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tgid.z() -; AKF_CHECK-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tgid_z ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR3:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tgid.z() @@ -111,14 +82,6 @@ define amdgpu_kernel void @use_tgid_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_x_z(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tgid_x_z -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.x() -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tgid.z() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tgid_x_z ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR3]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.x() @@ -135,14 +98,6 @@ define amdgpu_kernel void @use_tgid_x_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_y_z(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tgid_y_z -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tgid.z() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tgid_y_z ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR4:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.y() @@ -159,16 +114,6 @@ define amdgpu_kernel void @use_tgid_y_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tgid_x_y_z(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tgid_x_y_z -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.x() -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: [[VAL2:%.*]] = call i32 @llvm.r600.read.tgid.z() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL2]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tgid_x_y_z ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR4]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tgid.x() @@ -201,12 +146,6 @@ define amdgpu_kernel void @use_tidig_x(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_y(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tidig_y -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tidig.y() -; AKF_CHECK-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tidig_y ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR5:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tidig.y() @@ -219,12 +158,6 @@ define amdgpu_kernel void @use_tidig_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_z(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tidig_z -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tidig.z() -; AKF_CHECK-NEXT: store i32 [[VAL]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tidig_z ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR6:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL:%.*]] = call i32 @llvm.r600.read.tidig.z() @@ -253,14 +186,6 @@ define amdgpu_kernel void @use_tidig_x_tgid_x(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_y_tgid_y(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tidig_y_tgid_y -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tidig.y() -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tidig_y_tgid_y ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR7:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tidig.y() @@ -277,16 +202,6 @@ define amdgpu_kernel void @use_tidig_y_tgid_y(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_tidig_x_y_z(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_tidig_x_y_z -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tidig.x() -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tidig.y() -; AKF_CHECK-NEXT: [[VAL2:%.*]] = call i32 @llvm.r600.read.tidig.z() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL2]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_tidig_x_y_z ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR8:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tidig.x() @@ -307,22 +222,6 @@ define amdgpu_kernel void @use_tidig_x_y_z(ptr addrspace(1) %ptr) #1 { } define amdgpu_kernel void @use_all_workitems(ptr addrspace(1) %ptr) #1 { -; AKF_CHECK-LABEL: define {{[^@]+}}@use_all_workitems -; AKF_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR1]] { -; AKF_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tidig.x() -; AKF_CHECK-NEXT: [[VAL1:%.*]] = call i32 @llvm.r600.read.tidig.y() -; AKF_CHECK-NEXT: [[VAL2:%.*]] = call i32 @llvm.r600.read.tidig.z() -; AKF_CHECK-NEXT: [[VAL3:%.*]] = call i32 @llvm.r600.read.tgid.x() -; AKF_CHECK-NEXT: [[VAL4:%.*]] = call i32 @llvm.r600.read.tgid.y() -; AKF_CHECK-NEXT: [[VAL5:%.*]] = call i32 @llvm.r600.read.tgid.z() -; AKF_CHECK-NEXT: store volatile i32 [[VAL0]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL1]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL2]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL3]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL4]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: store volatile i32 [[VAL5]], ptr addrspace(1) [[PTR]], align 4 -; AKF_CHECK-NEXT: ret void -; ; ATTRIBUTOR_CHECK-LABEL: define {{[^@]+}}@use_all_workitems ; ATTRIBUTOR_CHECK-SAME: (ptr addrspace(1) [[PTR:%.*]]) #[[ATTR9:[0-9]+]] { ; ATTRIBUTOR_CHECK-NEXT: [[VAL0:%.*]] = call i32 @llvm.r600.read.tidig.x() @@ -394,8 +293,6 @@ attributes #0 = { nounwind readnone } attributes #1 = { nounwind } ;. -; AKF_CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } -; AKF_CHECK: attributes #[[ATTR1]] = { nounwind } ;. ; ATTRIBUTOR_CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } ; ATTRIBUTOR_CHECK: attributes #[[ATTR1]] = { nounwind "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } diff --git a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-flat-work-group-size.ll b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-flat-work-group-size.ll index fc13b86566f76..22cc5af30da66 100644 --- a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-flat-work-group-size.ll +++ b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-flat-work-group-size.ll @@ -35,9 +35,9 @@ entry: attributes #2 = {"amdgpu-flat-work-group-size"="128,128"} ; CHECK-LABEL: {{^}}min_1024_max_1024 -; CHECK: SGPRBlocks: 0 +; CHECK: SGPRBlocks: 2 ; CHECK: VGPRBlocks: 10 -; CHECK: NumSGPRsForWavesPerEU: 2{{$}} +; CHECK: NumSGPRsForWavesPerEU: 24{{$}} ; CHECK: NumVGPRsForWavesPerEU: 43 @var = addrspace(1) global float 0.0 define amdgpu_kernel void @min_1024_max_1024() #3 { diff --git a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-num-sgpr.ll b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-num-sgpr.ll index 46edf06c3b62c..d0107eb3ade27 100644 --- a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-num-sgpr.ll +++ b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-num-sgpr.ll @@ -4,8 +4,8 @@ ; ALL-LABEL: {{^}}max_10_sgprs: -; ALL: SGPRBlocks: 1 -; ALL: NumSGPRsForWavesPerEU: 10 +; ALL: SGPRBlocks: 2 +; ALL: NumSGPRsForWavesPerEU: 24 define amdgpu_kernel void @max_10_sgprs() #0 { %one = load volatile i32, ptr addrspace(4) poison %two = load volatile i32, ptr addrspace(4) poison diff --git a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll index 14519f5a5e77c..4507fd5865989 100644 --- a/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll +++ b/llvm/test/CodeGen/AMDGPU/attr-amdgpu-waves-per-eu.ll @@ -116,9 +116,9 @@ attributes #8 = {"amdgpu-waves-per-eu"="5,10"} ; Exactly 10 waves per execution unit. ; CHECK-LABEL: {{^}}exactly_10: -; CHECK: SGPRBlocks: 2 +; CHECK: SGPRBlocks: 3 ; CHECK: VGPRBlocks: 5 -; CHECK: NumSGPRsForWavesPerEU: 20 +; CHECK: NumSGPRsForWavesPerEU: 30 ; CHECK: NumVGPRsForWavesPerEU: 24 define amdgpu_kernel void @exactly_10() #9 { %val0 = load volatile float, ptr addrspace(1) @var diff --git a/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-globalisel.ll b/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-globalisel.ll index 682a57571d11e..35f0ccf5ba62f 100644 --- a/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-globalisel.ll +++ b/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-globalisel.ll @@ -392,7 +392,8 @@ define amdgpu_kernel void @call_use_intrinsic_workitem_id_x_cc_kernel() { ; GFX10: argumentInfo: ; GFX10-NEXT: privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' } ; GFX10-NEXT: kernargSegmentPtr: { reg: '$sgpr4_sgpr5' } -; GFX10-NEXT: workGroupIDX: { reg: '$sgpr6' } +; GFX10-NEXT: flatScratchInit: { reg: '$sgpr6_sgpr7' } +; GFX10-NEXT: workGroupIDX: { reg: '$sgpr8' } ; ; GFX10: name: call_without_private_to_flat_addrspacecast ; GFX10: argumentInfo: @@ -420,7 +421,8 @@ define amdgpu_kernel void @call_use_intrinsic_workitem_id_x_cc_kernel() { ; GFX10: argumentInfo: ; GFX10-NEXT: privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' } ; GFX10-NEXT: kernargSegmentPtr: { reg: '$sgpr4_sgpr5' } -; GFX10-NEXT: workGroupIDX: { reg: '$sgpr6' } +; GFX10-NEXT: flatScratchInit: { reg: '$sgpr6_sgpr7' } +; GFX10-NEXT: workGroupIDX: { reg: '$sgpr8' } ; ; GFX10: name: call_both_with_and_without_private_to_flat_addrspacecast ; GFX10: argumentInfo: @@ -434,7 +436,8 @@ define amdgpu_kernel void @call_use_intrinsic_workitem_id_x_cc_kernel() { ; GFX10: argumentInfo: ; GFX10-NEXT: privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' } ; GFX10-NEXT: kernargSegmentPtr: { reg: '$sgpr4_sgpr5' } -; GFX10-NEXT: workGroupIDX: { reg: '$sgpr6' } +; GFX10-NEXT: flatScratchInit: { reg: '$sgpr6_sgpr7' } +; GFX10-NEXT: workGroupIDX: { reg: '$sgpr8' } ; ; GFX10: name: call_call_without_private_to_flat_addrspacecast ; GFX10: argumentInfo: @@ -462,7 +465,8 @@ define amdgpu_kernel void @call_use_intrinsic_workitem_id_x_cc_kernel() { ; GFX10: argumentInfo: ; GFX10-NEXT: privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' } ; GFX10-NEXT: kernargSegmentPtr: { reg: '$sgpr4_sgpr5' } -; GFX10-NEXT: workGroupIDX: { reg: '$sgpr6' } +; GFX10-NEXT: flatScratchInit: { reg: '$sgpr6_sgpr7' } +; GFX10-NEXT: workGroupIDX: { reg: '$sgpr8' } ; ; GFX10: name: call_call_both_with_and_without_private_to_flat_addrspacecast ; GFX10: argumentInfo: @@ -476,7 +480,8 @@ define amdgpu_kernel void @call_use_intrinsic_workitem_id_x_cc_kernel() { ; GFX10: argumentInfo: ; GFX10-NEXT: privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' } ; GFX10-NEXT: kernargSegmentPtr: { reg: '$sgpr4_sgpr5' } -; GFX10-NEXT: workGroupIDX: { reg: '$sgpr6' } +; GFX10-NEXT: flatScratchInit: { reg: '$sgpr6_sgpr7' } +; GFX10-NEXT: workGroupIDX: { reg: '$sgpr8' } ; ; GFX10: name: with_cast_call_without_private_to_flat_addrspacecast ; GFX10: argumentInfo: @@ -490,7 +495,8 @@ define amdgpu_kernel void @call_use_intrinsic_workitem_id_x_cc_kernel() { ; GFX10: argumentInfo: ; GFX10-NEXT: privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' } ; GFX10-NEXT: kernargSegmentPtr: { reg: '$sgpr4_sgpr5' } -; GFX10-NEXT: workGroupIDX: { reg: '$sgpr6' } +; GFX10-NEXT: flatScratchInit: { reg: '$sgpr6_sgpr7' } +; GFX10-NEXT: workGroupIDX: { reg: '$sgpr8' } ; ; GFX10: name: with_cast_call_with_private_to_flat_addrspacecast ; GFX10: argumentInfo: @@ -504,7 +510,8 @@ define amdgpu_kernel void @call_use_intrinsic_workitem_id_x_cc_kernel() { ; GFX10: argumentInfo: ; GFX10-NEXT: privateSegmentBuffer: { reg: '$sgpr0_sgpr1_sgpr2_sgpr3' } ; GFX10-NEXT: kernargSegmentPtr: { reg: '$sgpr4_sgpr5' } -; GFX10-NEXT: workGroupIDX: { reg: '$sgpr6' } +; GFX10-NEXT: flatScratchInit: { reg: '$sgpr6_sgpr7' } +; GFX10-NEXT: workGroupIDX: { reg: '$sgpr8' } ; ; GFX10: name: with_indirect_call ; GFX10: argumentInfo: diff --git a/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior.ll b/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior.ll new file mode 100644 index 0000000000000..1b422252573db --- /dev/null +++ b/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior.ll @@ -0,0 +1,63 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 +; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -passes=amdgpu-attributor < %s | FileCheck -check-prefixes=GFX9 %s +; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -passes=amdgpu-attributor < %s | FileCheck -check-prefixes=GFX10 %s + +; +; None of these functions should have the attribute amdgpu-no-flat-scratch-init. In these tests +; we manually set the attribute for the functions. The purpose is to test how the amdgpu-attributor pass +; handles this situation. +; +;; tests of addrspacecast + +define void @with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) #0 { + %stof = addrspacecast ptr addrspace(5) %ptr to ptr + store volatile i32 0, ptr %stof + ret void +} + +define amdgpu_kernel void @with_private_to_flat_addrspacecast_cc_kernel(ptr addrspace(5) %ptr) #0 { + %stof = addrspacecast ptr addrspace(5) %ptr to ptr + store volatile i32 0, ptr %stof + ret void +} + +define void @call_with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) #0 { + call void @with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) + ret void +} + +define amdgpu_kernel void @call_with_private_to_flat_addrspacecast_cc_kernel(ptr addrspace(5) %ptr) #0 { + call void @with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) + ret void +} + +;; tests of addrspacecast in a constant + +define amdgpu_kernel void @private_constant_expression_use(ptr addrspace(1) nocapture %out) #0 { + store volatile ptr addrspacecast (ptr addrspace(5) inttoptr (i32 123 to ptr addrspace(5)) to ptr), ptr addrspace(1) %out, align 8 + ret void +} + +;; tests of intrinsics + +define amdgpu_kernel void @calls_intrin_ascast_cc_kernel(ptr addrspace(3) %ptr) #0 { + %1 = call ptr @llvm.amdgcn.addrspacecast.nonnull.p0.p3(ptr addrspace(3) %ptr) + store volatile i32 7, ptr %1, align 4 + ret void +} + +define void @calls_intrin_ascast(ptr addrspace(3) %ptr) #0 { + %1 = call ptr @llvm.amdgcn.addrspacecast.nonnull.p0.p3(ptr addrspace(3) %ptr) + store volatile i32 7, ptr %1, align 4 + ret void +} + +define amdgpu_kernel void @call_calls_intrin_ascast_cc_kernel(ptr addrspace(3) %ptr) #0 { + call void @calls_intrin_ascast(ptr addrspace(3) %ptr) + ret void +} + +attributes #0 = { "amdgpu-no-flat-scratch-init" } + +; GFX9: attributes #0 = { "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "target-cpu"="gfx900" "uniform-work-group-size"="false" } +; GFX10: attributes #0 = { "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "target-cpu"="gfx1010" "uniform-work-group-size"="false" } diff --git a/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior2.ll b/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior2.ll new file mode 100644 index 0000000000000..51caa84450ff3 --- /dev/null +++ b/llvm/test/CodeGen/AMDGPU/attributor-flatscratchinit-undefined-behavior2.ll @@ -0,0 +1,870 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 +; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 < %s | FileCheck -check-prefixes=GFX8 %s +; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx803 -mattr=+architected-flat-scratch < %s | FileCheck -check-prefixes=GFX8-ARCH-FLAT %s +; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck -check-prefixes=GFX9 %s +; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -mattr=+architected-flat-scratch < %s | FileCheck -check-prefixes=GFX9-ARCH-FLAT %s +; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx942 -mattr=+architected-flat-scratch < %s | FileCheck -check-prefixes=GFX942-ARCH-FLAT %s +; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 < %s | FileCheck -check-prefixes=GFX10 %s + +; +; None of these functions should have the attribute amdgpu-no-flat-scratch-init. In these tests +; we manually set the attribute for the functions. The purpose is to test how llc handles this. +; + +;; tests of addrspacecast + +define void @with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) #0 { +; GFX8-LABEL: with_private_to_flat_addrspacecast: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX8-NEXT: s_mov_b64 s[4:5], 0xc0 +; GFX8-NEXT: s_load_dword s4, s[4:5], 0x0 +; GFX8-NEXT: v_cmp_ne_u32_e32 vcc, -1, v0 +; GFX8-NEXT: v_mov_b32_e32 v2, 0 +; GFX8-NEXT: v_cndmask_b32_e32 v0, 0, v0, vcc +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: v_mov_b32_e32 v1, s4 +; GFX8-NEXT: v_cndmask_b32_e32 v1, 0, v1, vcc +; GFX8-NEXT: flat_store_dword v[0:1], v2 +; GFX8-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX8-NEXT: s_setpc_b64 s[30:31] +; +; GFX8-ARCH-FLAT-LABEL: with_private_to_flat_addrspacecast: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], 0xc0 +; GFX8-ARCH-FLAT-NEXT: s_load_dword s0, s[0:1], 0x0 +; GFX8-ARCH-FLAT-NEXT: v_cmp_ne_u32_e32 vcc, -1, v0 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX8-ARCH-FLAT-NEXT: v_cndmask_b32_e32 v0, 0, v0, vcc +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s0 +; GFX8-ARCH-FLAT-NEXT: v_cndmask_b32_e32 v1, 0, v1, vcc +; GFX8-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX9-LABEL: with_private_to_flat_addrspacecast: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX9-NEXT: s_mov_b64 s[4:5], src_private_base +; GFX9-NEXT: v_mov_b32_e32 v1, s5 +; GFX9-NEXT: v_cmp_ne_u32_e32 vcc, -1, v0 +; GFX9-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-NEXT: v_cndmask_b32_e32 v1, 0, v1, vcc +; GFX9-NEXT: v_cndmask_b32_e32 v0, 0, v0, vcc +; GFX9-NEXT: flat_store_dword v[0:1], v2 +; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX9-NEXT: s_setpc_b64 s[30:31] +; +; GFX9-ARCH-FLAT-LABEL: with_private_to_flat_addrspacecast: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_private_base +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX9-ARCH-FLAT-NEXT: v_cmp_ne_u32_e32 vcc, -1, v0 +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-ARCH-FLAT-NEXT: v_cndmask_b32_e32 v1, 0, v1, vcc +; GFX9-ARCH-FLAT-NEXT: v_cndmask_b32_e32 v0, 0, v0, vcc +; GFX9-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX942-ARCH-FLAT-LABEL: with_private_to_flat_addrspacecast: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_private_base +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX942-ARCH-FLAT-NEXT: v_cmp_ne_u32_e32 vcc, -1, v0 +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX942-ARCH-FLAT-NEXT: s_nop 0 +; GFX942-ARCH-FLAT-NEXT: v_cndmask_b32_e32 v1, 0, v1, vcc +; GFX942-ARCH-FLAT-NEXT: v_cndmask_b32_e32 v0, 0, v0, vcc +; GFX942-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 sc0 sc1 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX10-LABEL: with_private_to_flat_addrspacecast: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX10-NEXT: v_cmp_ne_u32_e32 vcc_lo, -1, v0 +; GFX10-NEXT: s_mov_b64 s[4:5], src_private_base +; GFX10-NEXT: v_mov_b32_e32 v2, 0 +; GFX10-NEXT: v_cndmask_b32_e64 v1, 0, s5, vcc_lo +; GFX10-NEXT: v_cndmask_b32_e32 v0, 0, v0, vcc_lo +; GFX10-NEXT: flat_store_dword v[0:1], v2 +; GFX10-NEXT: s_waitcnt_vscnt null, 0x0 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: s_setpc_b64 s[30:31] + %stof = addrspacecast ptr addrspace(5) %ptr to ptr + store volatile i32 0, ptr %stof + ret void +} + +define amdgpu_kernel void @with_private_to_flat_addrspacecast_cc_kernel(ptr addrspace(5) %ptr) #0 { +; GFX8-LABEL: with_private_to_flat_addrspacecast_cc_kernel: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_load_dword s0, s[8:9], 0x0 +; GFX8-NEXT: s_load_dword s1, s[8:9], 0xc8 +; GFX8-NEXT: v_mov_b32_e32 v2, 0 +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: s_cmp_lg_u32 s0, -1 +; GFX8-NEXT: s_cselect_b32 s1, s1, 0 +; GFX8-NEXT: s_cselect_b32 s0, s0, 0 +; GFX8-NEXT: v_mov_b32_e32 v0, s0 +; GFX8-NEXT: v_mov_b32_e32 v1, s1 +; GFX8-NEXT: flat_store_dword v[0:1], v2 +; GFX8-NEXT: s_waitcnt vmcnt(0) +; GFX8-NEXT: s_endpgm +; +; GFX8-ARCH-FLAT-LABEL: with_private_to_flat_addrspacecast_cc_kernel: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_load_dword s0, s[4:5], 0x0 +; GFX8-ARCH-FLAT-NEXT: s_load_dword s1, s[4:5], 0xc8 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_cmp_lg_u32 s0, -1 +; GFX8-ARCH-FLAT-NEXT: s_cselect_b32 s1, s1, 0 +; GFX8-ARCH-FLAT-NEXT: s_cselect_b32 s0, s0, 0 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s0 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX8-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_endpgm +; +; GFX9-LABEL: with_private_to_flat_addrspacecast_cc_kernel: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_load_dword s2, s[8:9], 0x0 +; GFX9-NEXT: s_mov_b64 s[0:1], src_private_base +; GFX9-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-NEXT: s_cmp_lg_u32 s2, -1 +; GFX9-NEXT: s_cselect_b32 s0, s1, 0 +; GFX9-NEXT: s_cselect_b32 s1, s2, 0 +; GFX9-NEXT: v_mov_b32_e32 v0, s1 +; GFX9-NEXT: v_mov_b32_e32 v1, s0 +; GFX9-NEXT: flat_store_dword v[0:1], v2 +; GFX9-NEXT: s_waitcnt vmcnt(0) +; GFX9-NEXT: s_endpgm +; +; GFX9-ARCH-FLAT-LABEL: with_private_to_flat_addrspacecast_cc_kernel: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_load_dword s2, s[4:5], 0x0 +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_private_base +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_cmp_lg_u32 s2, -1 +; GFX9-ARCH-FLAT-NEXT: s_cselect_b32 s0, s1, 0 +; GFX9-ARCH-FLAT-NEXT: s_cselect_b32 s1, s2, 0 +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s1 +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s0 +; GFX9-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_endpgm +; +; GFX942-ARCH-FLAT-LABEL: with_private_to_flat_addrspacecast_cc_kernel: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_load_dword s2, s[4:5], 0x0 +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_private_base +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_cmp_lg_u32 s2, -1 +; GFX942-ARCH-FLAT-NEXT: s_cselect_b32 s0, s1, 0 +; GFX942-ARCH-FLAT-NEXT: s_cselect_b32 s1, s2, 0 +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s1 +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s0 +; GFX942-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 sc0 sc1 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_endpgm +; +; GFX10-LABEL: with_private_to_flat_addrspacecast_cc_kernel: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_load_dword s2, s[8:9], 0x0 +; GFX10-NEXT: s_mov_b64 s[0:1], src_private_base +; GFX10-NEXT: v_mov_b32_e32 v2, 0 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: s_cmp_lg_u32 s2, -1 +; GFX10-NEXT: s_cselect_b32 s0, s2, 0 +; GFX10-NEXT: s_cselect_b32 s1, s1, 0 +; GFX10-NEXT: v_mov_b32_e32 v0, s0 +; GFX10-NEXT: v_mov_b32_e32 v1, s1 +; GFX10-NEXT: flat_store_dword v[0:1], v2 +; GFX10-NEXT: s_waitcnt_vscnt null, 0x0 +; GFX10-NEXT: s_endpgm + %stof = addrspacecast ptr addrspace(5) %ptr to ptr + store volatile i32 0, ptr %stof + ret void +} + +define void @call_with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) #0 { +; GFX8-LABEL: call_with_private_to_flat_addrspacecast: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX8-NEXT: s_mov_b32 s18, s33 +; GFX8-NEXT: s_mov_b32 s33, s32 +; GFX8-NEXT: s_xor_saveexec_b64 s[16:17], -1 +; GFX8-NEXT: buffer_store_dword v3, off, s[0:3], s33 ; 4-byte Folded Spill +; GFX8-NEXT: s_mov_b64 exec, s[16:17] +; GFX8-NEXT: s_addk_i32 s32, 0x400 +; GFX8-NEXT: s_getpc_b64 s[16:17] +; GFX8-NEXT: s_add_u32 s16, s16, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX8-NEXT: s_addc_u32 s17, s17, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX8-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX8-NEXT: v_writelane_b32 v3, s30, 0 +; GFX8-NEXT: v_writelane_b32 v3, s31, 1 +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX8-NEXT: v_readlane_b32 s31, v3, 1 +; GFX8-NEXT: v_readlane_b32 s30, v3, 0 +; GFX8-NEXT: s_mov_b32 s32, s33 +; GFX8-NEXT: s_xor_saveexec_b64 s[4:5], -1 +; GFX8-NEXT: buffer_load_dword v3, off, s[0:3], s33 ; 4-byte Folded Reload +; GFX8-NEXT: s_mov_b64 exec, s[4:5] +; GFX8-NEXT: s_mov_b32 s33, s18 +; GFX8-NEXT: s_waitcnt vmcnt(0) +; GFX8-NEXT: s_setpc_b64 s[30:31] +; +; GFX8-ARCH-FLAT-LABEL: call_with_private_to_flat_addrspacecast: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s2, s33 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s33, s32 +; GFX8-ARCH-FLAT-NEXT: s_xor_saveexec_b64 s[0:1], -1 +; GFX8-ARCH-FLAT-NEXT: s_add_i32 s3, s33, 8 +; GFX8-ARCH-FLAT-NEXT: scratch_store_dword off, v3, s3 ; 4-byte Folded Spill +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 exec, s[0:1] +; GFX8-ARCH-FLAT-NEXT: s_add_i32 s32, s32, 16 +; GFX8-ARCH-FLAT-NEXT: s_getpc_b64 s[0:1] +; GFX8-ARCH-FLAT-NEXT: s_add_u32 s0, s0, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX8-ARCH-FLAT-NEXT: s_addc_u32 s1, s1, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX8-ARCH-FLAT-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x0 +; GFX8-ARCH-FLAT-NEXT: v_writelane_b32 v3, s30, 0 +; GFX8-ARCH-FLAT-NEXT: v_writelane_b32 v3, s31, 1 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[0:1] +; GFX8-ARCH-FLAT-NEXT: v_readlane_b32 s31, v3, 1 +; GFX8-ARCH-FLAT-NEXT: v_readlane_b32 s30, v3, 0 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s32, s33 +; GFX8-ARCH-FLAT-NEXT: s_xor_saveexec_b64 s[0:1], -1 +; GFX8-ARCH-FLAT-NEXT: s_add_i32 s3, s33, 8 +; GFX8-ARCH-FLAT-NEXT: scratch_load_dword v3, off, s3 ; 4-byte Folded Reload +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 exec, s[0:1] +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s33, s2 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX9-LABEL: call_with_private_to_flat_addrspacecast: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX9-NEXT: s_mov_b32 s18, s33 +; GFX9-NEXT: s_mov_b32 s33, s32 +; GFX9-NEXT: s_xor_saveexec_b64 s[16:17], -1 +; GFX9-NEXT: buffer_store_dword v3, off, s[0:3], s33 ; 4-byte Folded Spill +; GFX9-NEXT: s_mov_b64 exec, s[16:17] +; GFX9-NEXT: s_addk_i32 s32, 0x400 +; GFX9-NEXT: s_getpc_b64 s[16:17] +; GFX9-NEXT: s_add_u32 s16, s16, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX9-NEXT: s_addc_u32 s17, s17, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX9-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX9-NEXT: v_writelane_b32 v3, s30, 0 +; GFX9-NEXT: v_writelane_b32 v3, s31, 1 +; GFX9-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX9-NEXT: v_readlane_b32 s31, v3, 1 +; GFX9-NEXT: v_readlane_b32 s30, v3, 0 +; GFX9-NEXT: s_mov_b32 s32, s33 +; GFX9-NEXT: s_xor_saveexec_b64 s[4:5], -1 +; GFX9-NEXT: buffer_load_dword v3, off, s[0:3], s33 ; 4-byte Folded Reload +; GFX9-NEXT: s_mov_b64 exec, s[4:5] +; GFX9-NEXT: s_mov_b32 s33, s18 +; GFX9-NEXT: s_waitcnt vmcnt(0) +; GFX9-NEXT: s_setpc_b64 s[30:31] +; +; GFX9-ARCH-FLAT-LABEL: call_with_private_to_flat_addrspacecast: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s2, s33 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s33, s32 +; GFX9-ARCH-FLAT-NEXT: s_xor_saveexec_b64 s[0:1], -1 +; GFX9-ARCH-FLAT-NEXT: scratch_store_dword off, v3, s33 ; 4-byte Folded Spill +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 exec, s[0:1] +; GFX9-ARCH-FLAT-NEXT: s_add_i32 s32, s32, 16 +; GFX9-ARCH-FLAT-NEXT: s_getpc_b64 s[0:1] +; GFX9-ARCH-FLAT-NEXT: s_add_u32 s0, s0, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX9-ARCH-FLAT-NEXT: s_addc_u32 s1, s1, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX9-ARCH-FLAT-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x0 +; GFX9-ARCH-FLAT-NEXT: v_writelane_b32 v3, s30, 0 +; GFX9-ARCH-FLAT-NEXT: v_writelane_b32 v3, s31, 1 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[0:1] +; GFX9-ARCH-FLAT-NEXT: v_readlane_b32 s31, v3, 1 +; GFX9-ARCH-FLAT-NEXT: v_readlane_b32 s30, v3, 0 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s32, s33 +; GFX9-ARCH-FLAT-NEXT: s_xor_saveexec_b64 s[0:1], -1 +; GFX9-ARCH-FLAT-NEXT: scratch_load_dword v3, off, s33 ; 4-byte Folded Reload +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 exec, s[0:1] +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s33, s2 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX942-ARCH-FLAT-LABEL: call_with_private_to_flat_addrspacecast: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s2, s33 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s33, s32 +; GFX942-ARCH-FLAT-NEXT: s_xor_saveexec_b64 s[0:1], -1 +; GFX942-ARCH-FLAT-NEXT: scratch_store_dword off, v3, s33 ; 4-byte Folded Spill +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 exec, s[0:1] +; GFX942-ARCH-FLAT-NEXT: s_add_i32 s32, s32, 16 +; GFX942-ARCH-FLAT-NEXT: s_getpc_b64 s[0:1] +; GFX942-ARCH-FLAT-NEXT: s_add_u32 s0, s0, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX942-ARCH-FLAT-NEXT: s_addc_u32 s1, s1, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX942-ARCH-FLAT-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x0 +; GFX942-ARCH-FLAT-NEXT: v_writelane_b32 v3, s30, 0 +; GFX942-ARCH-FLAT-NEXT: v_writelane_b32 v3, s31, 1 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[0:1] +; GFX942-ARCH-FLAT-NEXT: v_readlane_b32 s31, v3, 1 +; GFX942-ARCH-FLAT-NEXT: v_readlane_b32 s30, v3, 0 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s32, s33 +; GFX942-ARCH-FLAT-NEXT: s_xor_saveexec_b64 s[0:1], -1 +; GFX942-ARCH-FLAT-NEXT: scratch_load_dword v3, off, s33 ; 4-byte Folded Reload +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 exec, s[0:1] +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s33, s2 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX10-LABEL: call_with_private_to_flat_addrspacecast: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX10-NEXT: s_mov_b32 s18, s33 +; GFX10-NEXT: s_mov_b32 s33, s32 +; GFX10-NEXT: s_xor_saveexec_b32 s16, -1 +; GFX10-NEXT: buffer_store_dword v3, off, s[0:3], s33 ; 4-byte Folded Spill +; GFX10-NEXT: s_waitcnt_depctr 0xffe3 +; GFX10-NEXT: s_mov_b32 exec_lo, s16 +; GFX10-NEXT: s_addk_i32 s32, 0x200 +; GFX10-NEXT: s_getpc_b64 s[16:17] +; GFX10-NEXT: s_add_u32 s16, s16, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX10-NEXT: s_addc_u32 s17, s17, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX10-NEXT: v_writelane_b32 v3, s30, 0 +; GFX10-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX10-NEXT: v_writelane_b32 v3, s31, 1 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX10-NEXT: v_readlane_b32 s31, v3, 1 +; GFX10-NEXT: v_readlane_b32 s30, v3, 0 +; GFX10-NEXT: s_mov_b32 s32, s33 +; GFX10-NEXT: s_xor_saveexec_b32 s4, -1 +; GFX10-NEXT: buffer_load_dword v3, off, s[0:3], s33 ; 4-byte Folded Reload +; GFX10-NEXT: s_waitcnt_depctr 0xffe3 +; GFX10-NEXT: s_mov_b32 exec_lo, s4 +; GFX10-NEXT: s_mov_b32 s33, s18 +; GFX10-NEXT: s_waitcnt vmcnt(0) +; GFX10-NEXT: s_setpc_b64 s[30:31] + call void @with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) + ret void +} + +define amdgpu_kernel void @call_with_private_to_flat_addrspacecast_cc_kernel(ptr addrspace(5) %ptr) #0 { +; GFX8-LABEL: call_with_private_to_flat_addrspacecast_cc_kernel: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_add_u32 s0, s0, s15 +; GFX8-NEXT: s_addc_u32 s1, s1, 0 +; GFX8-NEXT: s_load_dword s15, s[8:9], 0x0 +; GFX8-NEXT: s_add_u32 s8, s8, 8 +; GFX8-NEXT: s_addc_u32 s9, s9, 0 +; GFX8-NEXT: s_getpc_b64 s[16:17] +; GFX8-NEXT: s_add_u32 s16, s16, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX8-NEXT: s_addc_u32 s17, s17, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX8-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX8-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX8-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX8-NEXT: v_or_b32_e32 v0, v0, v1 +; GFX8-NEXT: v_or_b32_e32 v31, v0, v2 +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: v_mov_b32_e32 v0, s15 +; GFX8-NEXT: s_mov_b32 s32, 0 +; GFX8-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX8-NEXT: s_endpgm +; +; GFX8-ARCH-FLAT-LABEL: call_with_private_to_flat_addrspacecast_cc_kernel: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s12, s8 +; GFX8-ARCH-FLAT-NEXT: s_add_u32 s8, s4, 8 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s13, s9 +; GFX8-ARCH-FLAT-NEXT: s_addc_u32 s9, s5, 0 +; GFX8-ARCH-FLAT-NEXT: s_load_dword s15, s[4:5], 0x0 +; GFX8-ARCH-FLAT-NEXT: s_getpc_b64 s[4:5] +; GFX8-ARCH-FLAT-NEXT: s_add_u32 s4, s4, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX8-ARCH-FLAT-NEXT: s_addc_u32 s5, s5, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX8-ARCH-FLAT-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0 +; GFX8-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX8-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX8-ARCH-FLAT-NEXT: v_or_b32_e32 v0, v0, v1 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s14, s10 +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[10:11], s[6:7] +; GFX8-ARCH-FLAT-NEXT: v_or_b32_e32 v31, v0, v2 +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[4:5], s[0:1] +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[6:7], s[2:3] +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s15 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s32, 0 +; GFX8-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX8-ARCH-FLAT-NEXT: s_endpgm +; +; GFX9-LABEL: call_with_private_to_flat_addrspacecast_cc_kernel: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_add_u32 s0, s0, s15 +; GFX9-NEXT: s_addc_u32 s1, s1, 0 +; GFX9-NEXT: s_load_dword s15, s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 s8, s8, 8 +; GFX9-NEXT: s_addc_u32 s9, s9, 0 +; GFX9-NEXT: s_getpc_b64 s[16:17] +; GFX9-NEXT: s_add_u32 s16, s16, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX9-NEXT: s_addc_u32 s17, s17, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX9-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX9-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX9-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX9-NEXT: v_or3_b32 v31, v0, v1, v2 +; GFX9-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-NEXT: v_mov_b32_e32 v0, s15 +; GFX9-NEXT: s_mov_b32 s32, 0 +; GFX9-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX9-NEXT: s_endpgm +; +; GFX9-ARCH-FLAT-LABEL: call_with_private_to_flat_addrspacecast_cc_kernel: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s12, s8 +; GFX9-ARCH-FLAT-NEXT: s_add_u32 s8, s4, 8 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s13, s9 +; GFX9-ARCH-FLAT-NEXT: s_addc_u32 s9, s5, 0 +; GFX9-ARCH-FLAT-NEXT: s_load_dword s15, s[4:5], 0x0 +; GFX9-ARCH-FLAT-NEXT: s_getpc_b64 s[4:5] +; GFX9-ARCH-FLAT-NEXT: s_add_u32 s4, s4, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX9-ARCH-FLAT-NEXT: s_addc_u32 s5, s5, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX9-ARCH-FLAT-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0 +; GFX9-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX9-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s14, s10 +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[10:11], s[6:7] +; GFX9-ARCH-FLAT-NEXT: v_or3_b32 v31, v0, v1, v2 +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[4:5], s[0:1] +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[6:7], s[2:3] +; GFX9-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s15 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s32, 0 +; GFX9-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX9-ARCH-FLAT-NEXT: s_endpgm +; +; GFX942-ARCH-FLAT-LABEL: call_with_private_to_flat_addrspacecast_cc_kernel: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s12, s8 +; GFX942-ARCH-FLAT-NEXT: s_add_u32 s8, s4, 8 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s13, s9 +; GFX942-ARCH-FLAT-NEXT: s_addc_u32 s9, s5, 0 +; GFX942-ARCH-FLAT-NEXT: s_load_dword s15, s[4:5], 0x0 +; GFX942-ARCH-FLAT-NEXT: s_getpc_b64 s[4:5] +; GFX942-ARCH-FLAT-NEXT: s_add_u32 s4, s4, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX942-ARCH-FLAT-NEXT: s_addc_u32 s5, s5, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX942-ARCH-FLAT-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s14, s10 +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[10:11], s[6:7] +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[4:5], s[0:1] +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[6:7], s[2:3] +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v31, v0 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s15 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s32, 0 +; GFX942-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX942-ARCH-FLAT-NEXT: s_endpgm +; +; GFX10-LABEL: call_with_private_to_flat_addrspacecast_cc_kernel: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s0, s0, s15 +; GFX10-NEXT: s_addc_u32 s1, s1, 0 +; GFX10-NEXT: s_load_dword s15, s[8:9], 0x0 +; GFX10-NEXT: s_add_u32 s8, s8, 8 +; GFX10-NEXT: s_addc_u32 s9, s9, 0 +; GFX10-NEXT: s_getpc_b64 s[16:17] +; GFX10-NEXT: s_add_u32 s16, s16, with_private_to_flat_addrspacecast@gotpcrel32@lo+4 +; GFX10-NEXT: s_addc_u32 s17, s17, with_private_to_flat_addrspacecast@gotpcrel32@hi+12 +; GFX10-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX10-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX10-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX10-NEXT: s_mov_b32 s32, 0 +; GFX10-NEXT: v_or3_b32 v31, v0, v1, v2 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: v_mov_b32_e32 v0, s15 +; GFX10-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX10-NEXT: s_endpgm + call void @with_private_to_flat_addrspacecast(ptr addrspace(5) %ptr) + ret void +} + +;; tests of addrspacecast in a constant + +define amdgpu_kernel void @private_constant_expression_use(ptr addrspace(1) nocapture %out) #0 { +; GFX8-LABEL: private_constant_expression_use: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_load_dword s2, s[8:9], 0xc8 +; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX8-NEXT: v_mov_b32_e32 v0, 0x7b +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: v_mov_b32_e32 v1, s2 +; GFX8-NEXT: v_mov_b32_e32 v3, s1 +; GFX8-NEXT: v_mov_b32_e32 v2, s0 +; GFX8-NEXT: flat_store_dwordx2 v[2:3], v[0:1] +; GFX8-NEXT: s_waitcnt vmcnt(0) +; GFX8-NEXT: s_endpgm +; +; GFX8-ARCH-FLAT-LABEL: private_constant_expression_use: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_load_dword s2, s[4:5], 0xc8 +; GFX8-ARCH-FLAT-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x0 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, 0x7b +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s2 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v3, s1 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, s0 +; GFX8-ARCH-FLAT-NEXT: flat_store_dwordx2 v[2:3], v[0:1] +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_endpgm +; +; GFX9-LABEL: private_constant_expression_use: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX9-NEXT: s_mov_b64 s[2:3], src_private_base +; GFX9-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-NEXT: v_mov_b32_e32 v0, 0x7b +; GFX9-NEXT: v_mov_b32_e32 v1, s3 +; GFX9-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-NEXT: global_store_dwordx2 v2, v[0:1], s[0:1] +; GFX9-NEXT: s_waitcnt vmcnt(0) +; GFX9-NEXT: s_endpgm +; +; GFX9-ARCH-FLAT-LABEL: private_constant_expression_use: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x0 +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[2:3], src_private_base +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, 0x7b +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s3 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: global_store_dwordx2 v2, v[0:1], s[0:1] +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_endpgm +; +; GFX942-ARCH-FLAT-LABEL: private_constant_expression_use: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x0 +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[2:3], src_private_base +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 0 +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, 0x7b +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s3 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: global_store_dwordx2 v2, v[0:1], s[0:1] sc0 sc1 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_endpgm +; +; GFX10-LABEL: private_constant_expression_use: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX10-NEXT: s_mov_b64 s[2:3], src_private_base +; GFX10-NEXT: v_mov_b32_e32 v2, 0 +; GFX10-NEXT: v_mov_b32_e32 v0, 0x7b +; GFX10-NEXT: v_mov_b32_e32 v1, s3 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: global_store_dwordx2 v2, v[0:1], s[0:1] +; GFX10-NEXT: s_waitcnt_vscnt null, 0x0 +; GFX10-NEXT: s_endpgm + store volatile ptr addrspacecast (ptr addrspace(5) inttoptr (i32 123 to ptr addrspace(5)) to ptr), ptr addrspace(1) %out, align 8 + ret void +} + +;; tests of intrinsics + +define amdgpu_kernel void @calls_intrin_ascast_cc_kernel(ptr addrspace(3) %ptr) #0 { +; GFX8-LABEL: calls_intrin_ascast_cc_kernel: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_load_dword s0, s[8:9], 0x0 +; GFX8-NEXT: s_load_dword s1, s[8:9], 0xcc +; GFX8-NEXT: v_mov_b32_e32 v2, 7 +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: v_mov_b32_e32 v0, s0 +; GFX8-NEXT: v_mov_b32_e32 v1, s1 +; GFX8-NEXT: flat_store_dword v[0:1], v2 +; GFX8-NEXT: s_waitcnt vmcnt(0) +; GFX8-NEXT: s_endpgm +; +; GFX8-ARCH-FLAT-LABEL: calls_intrin_ascast_cc_kernel: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_load_dword s0, s[4:5], 0x0 +; GFX8-ARCH-FLAT-NEXT: s_load_dword s1, s[4:5], 0xcc +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 7 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s0 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX8-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_endpgm +; +; GFX9-LABEL: calls_intrin_ascast_cc_kernel: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_load_dword s2, s[8:9], 0x0 +; GFX9-NEXT: s_mov_b64 s[0:1], src_shared_base +; GFX9-NEXT: v_mov_b32_e32 v1, s1 +; GFX9-NEXT: v_mov_b32_e32 v2, 7 +; GFX9-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-NEXT: v_mov_b32_e32 v0, s2 +; GFX9-NEXT: flat_store_dword v[0:1], v2 +; GFX9-NEXT: s_waitcnt vmcnt(0) +; GFX9-NEXT: s_endpgm +; +; GFX9-ARCH-FLAT-LABEL: calls_intrin_ascast_cc_kernel: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_load_dword s2, s[4:5], 0x0 +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_shared_base +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 7 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s2 +; GFX9-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_endpgm +; +; GFX942-ARCH-FLAT-LABEL: calls_intrin_ascast_cc_kernel: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_load_dword s2, s[4:5], 0x0 +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_shared_base +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 7 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s2 +; GFX942-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 sc0 sc1 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_endpgm +; +; GFX10-LABEL: calls_intrin_ascast_cc_kernel: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_load_dword s2, s[8:9], 0x0 +; GFX10-NEXT: s_mov_b64 s[0:1], src_shared_base +; GFX10-NEXT: v_mov_b32_e32 v2, 7 +; GFX10-NEXT: v_mov_b32_e32 v1, s1 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: v_mov_b32_e32 v0, s2 +; GFX10-NEXT: flat_store_dword v[0:1], v2 +; GFX10-NEXT: s_waitcnt_vscnt null, 0x0 +; GFX10-NEXT: s_endpgm + %1 = call ptr @llvm.amdgcn.addrspacecast.nonnull.p0.p3(ptr addrspace(3) %ptr) + store volatile i32 7, ptr %1, align 4 + ret void +} + +define void @calls_intrin_ascast(ptr addrspace(3) %ptr) #0 { +; GFX8-LABEL: calls_intrin_ascast: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX8-NEXT: s_mov_b64 s[4:5], 0xc4 +; GFX8-NEXT: s_load_dword s4, s[4:5], 0x0 +; GFX8-NEXT: v_mov_b32_e32 v2, 7 +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: v_mov_b32_e32 v1, s4 +; GFX8-NEXT: flat_store_dword v[0:1], v2 +; GFX8-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX8-NEXT: s_setpc_b64 s[30:31] +; +; GFX8-ARCH-FLAT-LABEL: calls_intrin_ascast: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], 0xc4 +; GFX8-ARCH-FLAT-NEXT: s_load_dword s0, s[0:1], 0x0 +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 7 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s0 +; GFX8-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX8-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX9-LABEL: calls_intrin_ascast: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX9-NEXT: s_mov_b64 s[4:5], src_shared_base +; GFX9-NEXT: v_mov_b32_e32 v1, s5 +; GFX9-NEXT: v_mov_b32_e32 v2, 7 +; GFX9-NEXT: flat_store_dword v[0:1], v2 +; GFX9-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX9-NEXT: s_setpc_b64 s[30:31] +; +; GFX9-ARCH-FLAT-LABEL: calls_intrin_ascast: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_shared_base +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 7 +; GFX9-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 +; GFX9-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX942-ARCH-FLAT-LABEL: calls_intrin_ascast: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[0:1], src_shared_base +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v1, s1 +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v2, 7 +; GFX942-ARCH-FLAT-NEXT: flat_store_dword v[0:1], v2 sc0 sc1 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: s_setpc_b64 s[30:31] +; +; GFX10-LABEL: calls_intrin_ascast: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) +; GFX10-NEXT: s_mov_b64 s[4:5], src_shared_base +; GFX10-NEXT: v_mov_b32_e32 v2, 7 +; GFX10-NEXT: v_mov_b32_e32 v1, s5 +; GFX10-NEXT: flat_store_dword v[0:1], v2 +; GFX10-NEXT: s_waitcnt_vscnt null, 0x0 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: s_setpc_b64 s[30:31] + %1 = call ptr @llvm.amdgcn.addrspacecast.nonnull.p0.p3(ptr addrspace(3) %ptr) + store volatile i32 7, ptr %1, align 4 + ret void +} + +define amdgpu_kernel void @call_calls_intrin_ascast_cc_kernel(ptr addrspace(3) %ptr) #0 { +; GFX8-LABEL: call_calls_intrin_ascast_cc_kernel: +; GFX8: ; %bb.0: +; GFX8-NEXT: s_add_u32 s0, s0, s15 +; GFX8-NEXT: s_addc_u32 s1, s1, 0 +; GFX8-NEXT: s_load_dword s15, s[8:9], 0x0 +; GFX8-NEXT: s_add_u32 s8, s8, 8 +; GFX8-NEXT: s_addc_u32 s9, s9, 0 +; GFX8-NEXT: s_getpc_b64 s[16:17] +; GFX8-NEXT: s_add_u32 s16, s16, calls_intrin_ascast@gotpcrel32@lo+4 +; GFX8-NEXT: s_addc_u32 s17, s17, calls_intrin_ascast@gotpcrel32@hi+12 +; GFX8-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX8-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX8-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX8-NEXT: v_or_b32_e32 v0, v0, v1 +; GFX8-NEXT: v_or_b32_e32 v31, v0, v2 +; GFX8-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-NEXT: v_mov_b32_e32 v0, s15 +; GFX8-NEXT: s_mov_b32 s32, 0 +; GFX8-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX8-NEXT: s_endpgm +; +; GFX8-ARCH-FLAT-LABEL: call_calls_intrin_ascast_cc_kernel: +; GFX8-ARCH-FLAT: ; %bb.0: +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s12, s8 +; GFX8-ARCH-FLAT-NEXT: s_add_u32 s8, s4, 8 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s13, s9 +; GFX8-ARCH-FLAT-NEXT: s_addc_u32 s9, s5, 0 +; GFX8-ARCH-FLAT-NEXT: s_load_dword s15, s[4:5], 0x0 +; GFX8-ARCH-FLAT-NEXT: s_getpc_b64 s[4:5] +; GFX8-ARCH-FLAT-NEXT: s_add_u32 s4, s4, calls_intrin_ascast@gotpcrel32@lo+4 +; GFX8-ARCH-FLAT-NEXT: s_addc_u32 s5, s5, calls_intrin_ascast@gotpcrel32@hi+12 +; GFX8-ARCH-FLAT-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0 +; GFX8-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX8-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX8-ARCH-FLAT-NEXT: v_or_b32_e32 v0, v0, v1 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s14, s10 +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[10:11], s[6:7] +; GFX8-ARCH-FLAT-NEXT: v_or_b32_e32 v31, v0, v2 +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[4:5], s[0:1] +; GFX8-ARCH-FLAT-NEXT: s_mov_b64 s[6:7], s[2:3] +; GFX8-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX8-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s15 +; GFX8-ARCH-FLAT-NEXT: s_mov_b32 s32, 0 +; GFX8-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX8-ARCH-FLAT-NEXT: s_endpgm +; +; GFX9-LABEL: call_calls_intrin_ascast_cc_kernel: +; GFX9: ; %bb.0: +; GFX9-NEXT: s_add_u32 s0, s0, s15 +; GFX9-NEXT: s_addc_u32 s1, s1, 0 +; GFX9-NEXT: s_load_dword s15, s[8:9], 0x0 +; GFX9-NEXT: s_add_u32 s8, s8, 8 +; GFX9-NEXT: s_addc_u32 s9, s9, 0 +; GFX9-NEXT: s_getpc_b64 s[16:17] +; GFX9-NEXT: s_add_u32 s16, s16, calls_intrin_ascast@gotpcrel32@lo+4 +; GFX9-NEXT: s_addc_u32 s17, s17, calls_intrin_ascast@gotpcrel32@hi+12 +; GFX9-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX9-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX9-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX9-NEXT: v_or3_b32 v31, v0, v1, v2 +; GFX9-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-NEXT: v_mov_b32_e32 v0, s15 +; GFX9-NEXT: s_mov_b32 s32, 0 +; GFX9-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX9-NEXT: s_endpgm +; +; GFX9-ARCH-FLAT-LABEL: call_calls_intrin_ascast_cc_kernel: +; GFX9-ARCH-FLAT: ; %bb.0: +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s12, s8 +; GFX9-ARCH-FLAT-NEXT: s_add_u32 s8, s4, 8 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s13, s9 +; GFX9-ARCH-FLAT-NEXT: s_addc_u32 s9, s5, 0 +; GFX9-ARCH-FLAT-NEXT: s_load_dword s15, s[4:5], 0x0 +; GFX9-ARCH-FLAT-NEXT: s_getpc_b64 s[4:5] +; GFX9-ARCH-FLAT-NEXT: s_add_u32 s4, s4, calls_intrin_ascast@gotpcrel32@lo+4 +; GFX9-ARCH-FLAT-NEXT: s_addc_u32 s5, s5, calls_intrin_ascast@gotpcrel32@hi+12 +; GFX9-ARCH-FLAT-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0 +; GFX9-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX9-ARCH-FLAT-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s14, s10 +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[10:11], s[6:7] +; GFX9-ARCH-FLAT-NEXT: v_or3_b32 v31, v0, v1, v2 +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[4:5], s[0:1] +; GFX9-ARCH-FLAT-NEXT: s_mov_b64 s[6:7], s[2:3] +; GFX9-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX9-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s15 +; GFX9-ARCH-FLAT-NEXT: s_mov_b32 s32, 0 +; GFX9-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX9-ARCH-FLAT-NEXT: s_endpgm +; +; GFX942-ARCH-FLAT-LABEL: call_calls_intrin_ascast_cc_kernel: +; GFX942-ARCH-FLAT: ; %bb.0: +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s12, s8 +; GFX942-ARCH-FLAT-NEXT: s_add_u32 s8, s4, 8 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s13, s9 +; GFX942-ARCH-FLAT-NEXT: s_addc_u32 s9, s5, 0 +; GFX942-ARCH-FLAT-NEXT: s_load_dword s15, s[4:5], 0x0 +; GFX942-ARCH-FLAT-NEXT: s_getpc_b64 s[4:5] +; GFX942-ARCH-FLAT-NEXT: s_add_u32 s4, s4, calls_intrin_ascast@gotpcrel32@lo+4 +; GFX942-ARCH-FLAT-NEXT: s_addc_u32 s5, s5, calls_intrin_ascast@gotpcrel32@hi+12 +; GFX942-ARCH-FLAT-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s14, s10 +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[10:11], s[6:7] +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[4:5], s[0:1] +; GFX942-ARCH-FLAT-NEXT: s_mov_b64 s[6:7], s[2:3] +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v31, v0 +; GFX942-ARCH-FLAT-NEXT: s_waitcnt lgkmcnt(0) +; GFX942-ARCH-FLAT-NEXT: v_mov_b32_e32 v0, s15 +; GFX942-ARCH-FLAT-NEXT: s_mov_b32 s32, 0 +; GFX942-ARCH-FLAT-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX942-ARCH-FLAT-NEXT: s_endpgm +; +; GFX10-LABEL: call_calls_intrin_ascast_cc_kernel: +; GFX10: ; %bb.0: +; GFX10-NEXT: s_add_u32 s0, s0, s15 +; GFX10-NEXT: s_addc_u32 s1, s1, 0 +; GFX10-NEXT: s_load_dword s15, s[8:9], 0x0 +; GFX10-NEXT: s_add_u32 s8, s8, 8 +; GFX10-NEXT: s_addc_u32 s9, s9, 0 +; GFX10-NEXT: s_getpc_b64 s[16:17] +; GFX10-NEXT: s_add_u32 s16, s16, calls_intrin_ascast@gotpcrel32@lo+4 +; GFX10-NEXT: s_addc_u32 s17, s17, calls_intrin_ascast@gotpcrel32@hi+12 +; GFX10-NEXT: v_lshlrev_b32_e32 v2, 20, v2 +; GFX10-NEXT: s_load_dwordx2 s[16:17], s[16:17], 0x0 +; GFX10-NEXT: v_lshlrev_b32_e32 v1, 10, v1 +; GFX10-NEXT: s_mov_b32 s32, 0 +; GFX10-NEXT: v_or3_b32 v31, v0, v1, v2 +; GFX10-NEXT: s_waitcnt lgkmcnt(0) +; GFX10-NEXT: v_mov_b32_e32 v0, s15 +; GFX10-NEXT: s_swappc_b64 s[30:31], s[16:17] +; GFX10-NEXT: s_endpgm + call void @calls_intrin_ascast(ptr addrspace(3) %ptr) + ret void +} + +attributes #0 = { "amdgpu-no-flat-scratch-init" } diff --git a/llvm/test/CodeGen/AMDGPU/attributor-noopt.ll b/llvm/test/CodeGen/AMDGPU/attributor-noopt.ll index 55ed11ac62972..4f341fa71cf68 100644 --- a/llvm/test/CodeGen/AMDGPU/attributor-noopt.ll +++ b/llvm/test/CodeGen/AMDGPU/attributor-noopt.ll @@ -30,9 +30,11 @@ ; NOOPT: .amdhsa_system_sgpr_workgroup_id_z 1 ; NOOPT: .amdhsa_system_sgpr_workgroup_info 0 ; NOOPT: .amdhsa_system_vgpr_workitem_id 2 -define amdgpu_kernel void @foo() { +define amdgpu_kernel void @foo() #0 { ret void } +attributes #0 = { "amdgpu-no-flat-scratch-init" } + !llvm.module.flags = !{!0} !0 = !{i32 1, !"amdhsa_code_object_version", i32 CODE_OBJECT_VERSION} diff --git a/llvm/test/CodeGen/AMDGPU/callee-special-input-sgprs-fixed-abi.ll b/llvm/test/CodeGen/AMDGPU/callee-special-input-sgprs-fixed-abi.ll index d5da3e00df1a6..10ca3c9d5f2c8 100644 --- a/llvm/test/CodeGen/AMDGPU/callee-special-input-sgprs-fixed-abi.ll +++ b/llvm/test/CodeGen/AMDGPU/callee-special-input-sgprs-fixed-abi.ll @@ -198,11 +198,11 @@ define hidden void @use_workgroup_id_yz() #1 { ; GCN-LABEL: {{^}}kern_indirect_use_workgroup_id_x: ; GCN-NOT: s6 -; GCN: s_getpc_b64 s[4:5] -; GCN-NEXT: s_add_u32 s4, s4, use_workgroup_id_x@rel32@lo+4 -; GCN-NEXT: s_addc_u32 s5, s5, use_workgroup_id_x@rel32@hi+12 +; GCN: s_getpc_b64 s[6:7] +; GCN-NEXT: s_add_u32 s6, s6, use_workgroup_id_x@rel32@lo+4 +; GCN-NEXT: s_addc_u32 s7, s7, use_workgroup_id_x@rel32@hi+12 ; GCN-NOT: s6 -; GCN: s_mov_b32 s12, s6 +; GCN: s_mov_b32 s12, s4 ; GCN: s_mov_b32 s32, 0 ; GCN: s_swappc_b64 ; GCN-NEXT: s_endpgm @@ -217,7 +217,7 @@ define amdgpu_kernel void @kern_indirect_use_workgroup_id_x() #1 { ; GCN-LABEL: {{^}}kern_indirect_use_workgroup_id_y: ; GCN-NOT: s12 -; GCN: s_mov_b32 s13, s7 +; GCN: s_mov_b32 s13, s5 ; GCN-NOT: s12 ; GCN: s_mov_b32 s32, 0 ; GCN: s_swappc_b64 @@ -233,7 +233,7 @@ define amdgpu_kernel void @kern_indirect_use_workgroup_id_y() #1 { ; GCN-LABEL: {{^}}kern_indirect_use_workgroup_id_z: ; GCN-NOT: s12 ; GCN-NOT: s13 -; GCN: s_mov_b32 s14, s7 +; GCN: s_mov_b32 s14, s5 ; GCN-NOT: s12 ; GCN-NOT: s13 @@ -250,8 +250,8 @@ define amdgpu_kernel void @kern_indirect_use_workgroup_id_z() #1 { ; GCN-LABEL: {{^}}kern_indirect_use_workgroup_id_xy: ; GCN-NOT: s14 -; GCN: s_mov_b32 s12, s6 -; GCN-NEXT: s_mov_b32 s13, s7 +; GCN: s_mov_b32 s12, s4 +; GCN-NEXT: s_mov_b32 s13, s5 ; GCN-NOT: s14 ; GCN: s_mov_b32 s32, 0 @@ -266,9 +266,9 @@ define amdgpu_kernel void @kern_indirect_use_workgroup_id_xy() #1 { } ; GCN-LABEL: {{^}}kern_indirect_use_workgroup_id_xyz: -; GCN: s_mov_b32 s12, s6 -; GCN: s_mov_b32 s13, s7 -; GCN: s_mov_b32 s14, s8 +; GCN: s_mov_b32 s12, s4 +; GCN: s_mov_b32 s13, s5 +; GCN: s_mov_b32 s14, s6 ; GCN: s_mov_b32 s32, 0 ; GCN: s_swappc_b64 @@ -283,8 +283,8 @@ define amdgpu_kernel void @kern_indirect_use_workgroup_id_xyz() #1 { ; GCN-LABEL: {{^}}kern_indirect_use_workgroup_id_xz: ; GCN-NOT: s13 -; GCN: s_mov_b32 s12, s6 -; GCN-NEXT: s_mov_b32 s14, s7 +; GCN: s_mov_b32 s12, s4 +; GCN-NEXT: s_mov_b32 s14, s5 ; GCN-NOT: s13 ; GCN: s_mov_b32 s32, 0 @@ -300,8 +300,8 @@ define amdgpu_kernel void @kern_indirect_use_workgroup_id_xz() #1 { ; GCN-LABEL: {{^}}kern_indirect_use_workgroup_id_yz: -; GCN: s_mov_b32 s13, s7 -; GCN: s_mov_b32 s14, s8 +; GCN: s_mov_b32 s13, s5 +; GCN: s_mov_b32 s14, s6 ; GCN: s_mov_b32 s32, 0 ; GCN: s_swappc_b64 @@ -382,7 +382,7 @@ define hidden void @other_arg_use_workgroup_id_z(i32 %arg0) #1 { ; GCN-NOT: s13 ; GCN-NOT: s14 -; GCN-DAG: s_mov_b32 s12, s6 +; GCN-DAG: s_mov_b32 s12, s4 ; GCN-DAG: v_mov_b32_e32 v0, 0x22b ; GCN-NOT: s13 ; GCN-NOT: s14 @@ -400,7 +400,7 @@ define amdgpu_kernel void @kern_indirect_other_arg_use_workgroup_id_x() #1 { ; GCN-LABEL: {{^}}kern_indirect_other_arg_use_workgroup_id_y: ; GCN-DAG: v_mov_b32_e32 v0, 0x22b -; GCN-DAG: s_mov_b32 s13, s7 +; GCN-DAG: s_mov_b32 s13, s5 ; GCN-DAG: s_mov_b32 s32, 0 ; GCN: s_swappc_b64 @@ -415,7 +415,7 @@ define amdgpu_kernel void @kern_indirect_other_arg_use_workgroup_id_y() #1 { ; GCN-LABEL: {{^}}kern_indirect_other_arg_use_workgroup_id_z: ; GCN-DAG: v_mov_b32_e32 v0, 0x22b -; GCN-DAG: s_mov_b32 s14, s7 +; GCN-DAG: s_mov_b32 s14, s5 ; GCN: s_mov_b32 s32, 0 ; GCN: s_swappc_b64 @@ -474,7 +474,7 @@ define hidden void @use_every_sgpr_input() #1 { ; GCN: .amdhsa_user_sgpr_queue_ptr 1 ; GCN: .amdhsa_user_sgpr_kernarg_segment_ptr 1 ; GCN: .amdhsa_user_sgpr_dispatch_id 1 -; GCN: .amdhsa_user_sgpr_flat_scratch_init 1 +; GCN: .amdhsa_user_sgpr_flat_scratch_init 0 ; GCN: .amdhsa_user_sgpr_private_segment_size 0 ; GCN: .amdhsa_system_sgpr_private_segment_wavefront_offset 1 ; GCN: .amdhsa_system_sgpr_workgroup_id_x 1 @@ -499,7 +499,7 @@ define amdgpu_kernel void @kern_indirect_use_every_sgpr_input(i8) #1 { ; GCN: .amdhsa_user_sgpr_queue_ptr 1 ; GCN: .amdhsa_user_sgpr_kernarg_segment_ptr 0 ; GCN: .amdhsa_user_sgpr_dispatch_id 1 -; GCN: .amdhsa_user_sgpr_flat_scratch_init 1 +; GCN: .amdhsa_user_sgpr_flat_scratch_init 0 ; GCN: .amdhsa_user_sgpr_private_segment_size 0 ; GCN: .amdhsa_system_sgpr_private_segment_wavefront_offset 1 ; GCN: .amdhsa_system_sgpr_workgroup_id_x 1 diff --git a/llvm/test/CodeGen/AMDGPU/code-object-v3.ll b/llvm/test/CodeGen/AMDGPU/code-object-v3.ll index ee4a2ed883b63..3fe3cafd729a7 100644 --- a/llvm/test/CodeGen/AMDGPU/code-object-v3.ll +++ b/llvm/test/CodeGen/AMDGPU/code-object-v3.ll @@ -68,7 +68,7 @@ define amdgpu_kernel void @fadd( ptr addrspace(1) %r, ptr addrspace(1) %a, - ptr addrspace(1) %b) { + ptr addrspace(1) %b) #0 { entry: %a.val = load float, ptr addrspace(1) %a %b.val = load float, ptr addrspace(1) %b @@ -80,7 +80,7 @@ entry: define amdgpu_kernel void @fsub( ptr addrspace(1) %r, ptr addrspace(1) %a, - ptr addrspace(1) %b) { + ptr addrspace(1) %b) #0 { entry: %a.val = load float, ptr addrspace(1) %a %b.val = load float, ptr addrspace(1) %b @@ -99,7 +99,9 @@ define amdgpu_kernel void @empty( i32 %i, ptr addrspace(1) %r, ptr addrspace(1) %a, - ptr addrspace(1) %b) { + ptr addrspace(1) %b) #0 { entry: ret void } + +attributes #0 = { "amdgpu-no-flat-scratch-init" } diff --git a/llvm/test/CodeGen/AMDGPU/combine-reg-or-const.ll b/llvm/test/CodeGen/AMDGPU/combine-reg-or-const.ll index c17cf1cd6bca4..c167834470e3b 100644 --- a/llvm/test/CodeGen/AMDGPU/combine-reg-or-const.ll +++ b/llvm/test/CodeGen/AMDGPU/combine-reg-or-const.ll @@ -5,6 +5,9 @@ define protected amdgpu_kernel void @_Z11test_kernelPii(ptr addrspace(1) nocapture %Ad.coerce, i32 %s) local_unnamed_addr #5 { ; CHECK-LABEL: _Z11test_kernelPii: ; CHECK: ; %bb.0: ; %entry +; CHECK-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-NEXT: s_add_i32 s12, s12, s17 +; CHECK-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-NEXT: s_load_dword s0, s[8:9], 0x2 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: s_cmp_lg_u32 s0, 3 diff --git a/llvm/test/CodeGen/AMDGPU/dagcomb-extract-vec-elt-different-sizes.ll b/llvm/test/CodeGen/AMDGPU/dagcomb-extract-vec-elt-different-sizes.ll index fcb8fa5997b7e..fc17d9288bf40 100644 --- a/llvm/test/CodeGen/AMDGPU/dagcomb-extract-vec-elt-different-sizes.ll +++ b/llvm/test/CodeGen/AMDGPU/dagcomb-extract-vec-elt-different-sizes.ll @@ -6,6 +6,8 @@ define amdgpu_kernel void @eggs(i1 %arg, ptr addrspace(1) %arg1, ptr %arg2, ptr %arg3, ptr %arg4, ptr %arg5, ptr %arg6, ptr %arg7, ptr %arg8, ptr %arg9) { ; CHECK-LABEL: eggs: ; CHECK: ; %bb.0: ; %bb +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; CHECK-NEXT: s_load_dword s0, s[8:9], 0x0 ; CHECK-NEXT: s_load_dwordx16 s[12:27], s[8:9], 0x8 ; CHECK-NEXT: v_mov_b32_e32 v0, 0 diff --git a/llvm/test/CodeGen/AMDGPU/duplicate-attribute-indirect.ll b/llvm/test/CodeGen/AMDGPU/duplicate-attribute-indirect.ll index 39554e05c96b4..f964170ccdda5 100644 --- a/llvm/test/CodeGen/AMDGPU/duplicate-attribute-indirect.ll +++ b/llvm/test/CodeGen/AMDGPU/duplicate-attribute-indirect.ll @@ -1,11 +1,7 @@ ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals -; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -amdgpu-annotate-kernel-features %s | FileCheck -check-prefix=AKF_GCN %s ; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -passes=amdgpu-attributor %s | FileCheck -check-prefix=ATTRIBUTOR_GCN %s define internal void @indirect() { -; AKF_GCN-LABEL: define {{[^@]+}}@indirect() { -; AKF_GCN-NEXT: ret void -; ; ATTRIBUTOR_GCN-LABEL: define {{[^@]+}}@indirect ; ATTRIBUTOR_GCN-SAME: () #[[ATTR0:[0-9]+]] { ; ATTRIBUTOR_GCN-NEXT: ret void @@ -14,14 +10,6 @@ define internal void @indirect() { } define amdgpu_kernel void @test_simple_indirect_call() #0 { -; AKF_GCN-LABEL: define {{[^@]+}}@test_simple_indirect_call -; AKF_GCN-SAME: () #[[ATTR0:[0-9]+]] { -; AKF_GCN-NEXT: [[FPTR:%.*]] = alloca ptr, align 8, addrspace(5) -; AKF_GCN-NEXT: store ptr @indirect, ptr addrspace(5) [[FPTR]], align 8 -; AKF_GCN-NEXT: [[FP:%.*]] = load ptr, ptr addrspace(5) [[FPTR]], align 8 -; AKF_GCN-NEXT: call void [[FP]]() -; AKF_GCN-NEXT: ret void -; ; ATTRIBUTOR_GCN-LABEL: define {{[^@]+}}@test_simple_indirect_call ; ATTRIBUTOR_GCN-SAME: () #[[ATTR1:[0-9]+]] { ; ATTRIBUTOR_GCN-NEXT: [[FPTR:%.*]] = alloca ptr, align 8, addrspace(5) @@ -40,7 +28,6 @@ define amdgpu_kernel void @test_simple_indirect_call() #0 { attributes #0 = { "amdgpu-no-dispatch-id" } ;. -; AKF_GCN: attributes #[[ATTR0]] = { "amdgpu-calls" "amdgpu-no-dispatch-id" "amdgpu-stack-objects" } ;. ; ATTRIBUTOR_GCN: attributes #[[ATTR0]] = { "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } ; ATTRIBUTOR_GCN: attributes #[[ATTR1]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } diff --git a/llvm/test/CodeGen/AMDGPU/exec-mask-opt-cannot-create-empty-or-backward-segment.ll b/llvm/test/CodeGen/AMDGPU/exec-mask-opt-cannot-create-empty-or-backward-segment.ll index 9104dc68eb9b4..72913d2596ebf 100644 --- a/llvm/test/CodeGen/AMDGPU/exec-mask-opt-cannot-create-empty-or-backward-segment.ll +++ b/llvm/test/CodeGen/AMDGPU/exec-mask-opt-cannot-create-empty-or-backward-segment.ll @@ -9,7 +9,7 @@ define amdgpu_kernel void @cannot_create_empty_or_backwards_segment(i1 %arg, i1 ; CHECK-NEXT: s_load_dword s2, s[8:9], 0x0 ; CHECK-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CHECK-NEXT: s_load_dword s6, s[8:9], 0x4 -; CHECK-NEXT: s_add_u32 s24, s24, s15 +; CHECK-NEXT: s_add_u32 s24, s24, s17 ; CHECK-NEXT: s_addc_u32 s25, s25, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: s_bitcmp1_b32 s2, 0 diff --git a/llvm/test/CodeGen/AMDGPU/expand-scalar-carry-out-select-user.ll b/llvm/test/CodeGen/AMDGPU/expand-scalar-carry-out-select-user.ll index f3aec696abdee..e6f02295e67d5 100644 --- a/llvm/test/CodeGen/AMDGPU/expand-scalar-carry-out-select-user.ll +++ b/llvm/test/CodeGen/AMDGPU/expand-scalar-carry-out-select-user.ll @@ -94,6 +94,9 @@ define amdgpu_kernel void @s_add_co_br_user(i32 %i) { ; GFX7-LABEL: s_add_co_br_user: ; GFX7: ; %bb.0: ; %bb ; GFX7-NEXT: s_load_dword s2, s[8:9], 0x0 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_add_i32 s0, s2, s2 ; GFX7-NEXT: s_cmp_lt_u32 s0, s2 diff --git a/llvm/test/CodeGen/AMDGPU/extract_vector_elt-i8.ll b/llvm/test/CodeGen/AMDGPU/extract_vector_elt-i8.ll index 0c25ca5076790..fac9f5bf826a6 100644 --- a/llvm/test/CodeGen/AMDGPU/extract_vector_elt-i8.ll +++ b/llvm/test/CodeGen/AMDGPU/extract_vector_elt-i8.ll @@ -5,6 +5,9 @@ define amdgpu_kernel void @extract_vector_elt_v1i8(ptr addrspace(1) %out, <1 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v1i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -18,6 +21,9 @@ define amdgpu_kernel void @extract_vector_elt_v1i8(ptr addrspace(1) %out, <1 x i ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -32,6 +38,9 @@ define amdgpu_kernel void @extract_vector_elt_v1i8(ptr addrspace(1) %out, <1 x i define amdgpu_kernel void @extract_vector_elt_v2i8(ptr addrspace(1) %out, <2 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v2i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -54,6 +63,9 @@ define amdgpu_kernel void @extract_vector_elt_v2i8(ptr addrspace(1) %out, <2 x i ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 8 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -80,6 +92,9 @@ define amdgpu_kernel void @extract_vector_elt_v2i8(ptr addrspace(1) %out, <2 x i define amdgpu_kernel void @extract_vector_elt_v3i8(ptr addrspace(1) %out, <3 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v3i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -102,6 +117,9 @@ define amdgpu_kernel void @extract_vector_elt_v3i8(ptr addrspace(1) %out, <3 x i ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -128,6 +146,9 @@ define amdgpu_kernel void @extract_vector_elt_v3i8(ptr addrspace(1) %out, <3 x i define amdgpu_kernel void @extract_vector_elt_v4i8(ptr addrspace(1) %out, <4 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v4i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -150,6 +171,9 @@ define amdgpu_kernel void @extract_vector_elt_v4i8(ptr addrspace(1) %out, <4 x i ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -176,6 +200,9 @@ define amdgpu_kernel void @extract_vector_elt_v4i8(ptr addrspace(1) %out, <4 x i define amdgpu_kernel void @extract_vector_elt_v8i8(<8 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v8i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dword s0, s[8:9], 0x0 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_lshr_b32 s1, s0, 16 @@ -192,10 +219,13 @@ define amdgpu_kernel void @extract_vector_elt_v8i8(<8 x i8> %foo) #0 { ; VI-LABEL: extract_vector_elt_v8i8: ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s0, s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 0 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s1, s0, 16 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_byte v[0:1], v3 @@ -213,6 +243,9 @@ define amdgpu_kernel void @extract_vector_elt_v8i8(<8 x i8> %foo) #0 { define amdgpu_kernel void @extract_vector_elt_v16i8(ptr addrspace(1) %out, <16 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v16i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; SI-NEXT: s_load_dword s2, s[8:9], 0x4 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -235,6 +268,9 @@ define amdgpu_kernel void @extract_vector_elt_v16i8(ptr addrspace(1) %out, <16 x ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x10 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -261,6 +297,9 @@ define amdgpu_kernel void @extract_vector_elt_v16i8(ptr addrspace(1) %out, <16 x define amdgpu_kernel void @extract_vector_elt_v32i8(<32 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v32i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dword s0, s[8:9], 0x0 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_lshr_b32 s1, s0, 16 @@ -277,10 +316,13 @@ define amdgpu_kernel void @extract_vector_elt_v32i8(<32 x i8> %foo) #0 { ; VI-LABEL: extract_vector_elt_v32i8: ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s0, s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 0 -; VI-NEXT: v_mov_b32_e32 v1, 0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s1, s0, 16 +; VI-NEXT: v_mov_b32_e32 v1, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_byte v[0:1], v3 @@ -298,6 +340,9 @@ define amdgpu_kernel void @extract_vector_elt_v32i8(<32 x i8> %foo) #0 { define amdgpu_kernel void @extract_vector_elt_v64i8(ptr addrspace(1) %out, <64 x i8> %foo) #0 { ; SI-LABEL: extract_vector_elt_v64i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; SI-NEXT: s_load_dword s2, s[8:9], 0x10 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -320,6 +365,9 @@ define amdgpu_kernel void @extract_vector_elt_v64i8(ptr addrspace(1) %out, <64 x ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x40 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -351,6 +399,9 @@ define amdgpu_kernel void @extract_vector_elt_v64i8(ptr addrspace(1) %out, <64 x define amdgpu_kernel void @dynamic_extract_vector_elt_v2i8(ptr addrspace(1) %out, [8 x i32], <2 x i8> %foo, [8 x i32], i32 %idx) #0 { ; SI-LABEL: dynamic_extract_vector_elt_v2i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dword s2, s[8:9], 0xa ; SI-NEXT: s_load_dword s3, s[8:9], 0x13 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -370,11 +421,14 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v2i8(ptr addrspace(1) %out ; VI-NEXT: s_load_dword s2, s[8:9], 0x4c ; VI-NEXT: s_load_dword s3, s[8:9], 0x28 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshl_b32 s2, s2, 3 ; VI-NEXT: s_and_b32 s3, s3, 0xffff ; VI-NEXT: s_lshr_b32 s2, s3, s2 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: flat_store_byte v[0:1], v2 @@ -388,6 +442,9 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v2i8(ptr addrspace(1) %out define amdgpu_kernel void @dynamic_extract_vector_elt_v3i8(ptr addrspace(1) %out, [8 x i32], <3 x i8> %foo, [8 x i32], i32 %idx) #0 { ; SI-LABEL: dynamic_extract_vector_elt_v3i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dword s2, s[8:9], 0x13 ; SI-NEXT: s_load_dword s3, s[8:9], 0xa ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -406,10 +463,13 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v3i8(ptr addrspace(1) %out ; VI-NEXT: s_load_dword s2, s[8:9], 0x4c ; VI-NEXT: s_load_dword s3, s[8:9], 0x28 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshl_b32 s2, s2, 3 ; VI-NEXT: s_lshr_b32 s2, s3, s2 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: flat_store_byte v[0:1], v2 @@ -424,6 +484,9 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v3i8(ptr addrspace(1) %out define amdgpu_kernel void @dynamic_extract_vector_elt_v4i8(ptr addrspace(1) %out, ptr addrspace(4) %vec.ptr, [8 x i32], i32 %idx) #0 { ; SI-LABEL: dynamic_extract_vector_elt_v4i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; SI-NEXT: s_load_dword s4, s[8:9], 0xc ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -442,6 +505,9 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v4i8(ptr addrspace(1) %out ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -463,6 +529,9 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v4i8(ptr addrspace(1) %out define amdgpu_kernel void @dynamic_extract_vector_elt_v8i8(ptr addrspace(1) %out, ptr addrspace(4) %vec.ptr, i32 %idx) #0 { ; SI-LABEL: dynamic_extract_vector_elt_v8i8: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; SI-NEXT: s_load_dword s4, s[8:9], 0x4 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -481,6 +550,9 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v8i8(ptr addrspace(1) %out ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -502,6 +574,9 @@ define amdgpu_kernel void @dynamic_extract_vector_elt_v8i8(ptr addrspace(1) %out define amdgpu_kernel void @reduce_load_vector_v8i8_extract_0123() #0 { ; SI-LABEL: reduce_load_vector_v8i8_extract_0123: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_mov_b64 s[0:1], 0 ; SI-NEXT: s_load_dword s0, s[0:1], 0x0 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -526,6 +601,9 @@ define amdgpu_kernel void @reduce_load_vector_v8i8_extract_0123() #0 { ; VI: ; %bb.0: ; VI-NEXT: s_mov_b64 s[0:1], 0 ; VI-NEXT: s_load_dword s0, s[0:1], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s1, s0, 8 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -558,6 +636,9 @@ define amdgpu_kernel void @reduce_load_vector_v8i8_extract_0123() #0 { define amdgpu_kernel void @reduce_load_vector_v8i8_extract_0145() #0 { ; SI-LABEL: reduce_load_vector_v8i8_extract_0145: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_mov_b64 s[0:1], 0 ; SI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x0 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -581,6 +662,9 @@ define amdgpu_kernel void @reduce_load_vector_v8i8_extract_0145() #0 { ; VI: ; %bb.0: ; VI-NEXT: s_mov_b64 s[0:1], 0 ; VI-NEXT: s_load_dwordx2 s[0:1], s[0:1], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s2, s0, 8 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -612,6 +696,9 @@ define amdgpu_kernel void @reduce_load_vector_v8i8_extract_0145() #0 { define amdgpu_kernel void @reduce_load_vector_v8i8_extract_45() #0 { ; SI-LABEL: reduce_load_vector_v8i8_extract_45: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_mov_b64 s[0:1], 4 ; SI-NEXT: s_load_dword s0, s[0:1], 0x0 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -628,6 +715,9 @@ define amdgpu_kernel void @reduce_load_vector_v8i8_extract_45() #0 { ; VI: ; %bb.0: ; VI-NEXT: s_mov_b64 s[0:1], 4 ; VI-NEXT: s_load_dword s0, s[0:1], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s1, s0, 8 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -649,6 +739,9 @@ define amdgpu_kernel void @reduce_load_vector_v8i8_extract_45() #0 { define amdgpu_kernel void @reduce_load_vector_v16i8_extract_0145() #0 { ; SI-LABEL: reduce_load_vector_v16i8_extract_0145: ; SI: ; %bb.0: +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_mov_b64 s[0:1], 0 ; SI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0 ; SI-NEXT: s_waitcnt lgkmcnt(0) @@ -672,6 +765,9 @@ define amdgpu_kernel void @reduce_load_vector_v16i8_extract_0145() #0 { ; VI: ; %bb.0: ; VI-NEXT: s_mov_b64 s[0:1], 0 ; VI-NEXT: s_load_dwordx4 s[0:3], s[0:1], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s2, s0, 8 ; VI-NEXT: v_mov_b32_e32 v0, s0 diff --git a/llvm/test/CodeGen/AMDGPU/fabs.f16.ll b/llvm/test/CodeGen/AMDGPU/fabs.f16.ll index 32f75f3835226..7b6a363c42708 100644 --- a/llvm/test/CodeGen/AMDGPU/fabs.f16.ll +++ b/llvm/test/CodeGen/AMDGPU/fabs.f16.ll @@ -14,6 +14,9 @@ define amdgpu_kernel void @s_fabs_free_f16(ptr addrspace(1) %out, i16 %in) { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_and_b32 s2, s2, 0x7fff ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -26,6 +29,9 @@ define amdgpu_kernel void @s_fabs_free_f16(ptr addrspace(1) %out, i16 %in) { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_and_b32 s2, s2, 0x7fff ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -80,6 +86,9 @@ define amdgpu_kernel void @s_fabs_f16(ptr addrspace(1) %out, half %in) { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_and_b32 s2, s2, 0x7fff ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -92,6 +101,9 @@ define amdgpu_kernel void @s_fabs_f16(ptr addrspace(1) %out, half %in) { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_and_b32 s2, s2, 0x7fff ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -145,6 +157,9 @@ define amdgpu_kernel void @s_fabs_v2f16(ptr addrspace(1) %out, <2 x half> %in) { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_and_b32 s2, s2, 0x7fff7fff ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -157,6 +172,9 @@ define amdgpu_kernel void @s_fabs_v2f16(ptr addrspace(1) %out, <2 x half> %in) { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_and_b32 s2, s2, 0x7fff7fff ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -196,6 +214,9 @@ define amdgpu_kernel void @s_fabs_v4f16(ptr addrspace(1) %out, <4 x half> %in) { ; CI-LABEL: s_fabs_v4f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_and_b32 s3, s3, 0x7fff7fff ; CI-NEXT: s_and_b32 s2, s2, 0x7fff7fff @@ -209,6 +230,9 @@ define amdgpu_kernel void @s_fabs_v4f16(ptr addrspace(1) %out, <4 x half> %in) { ; VI-LABEL: s_fabs_v4f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_and_b32 s3, s3, 0x7fff7fff ; VI-NEXT: s_and_b32 s2, s2, 0x7fff7fff @@ -251,6 +275,9 @@ define amdgpu_kernel void @fabs_fold_f16(ptr addrspace(1) %out, half %in0, half ; CI-LABEL: fabs_fold_f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f32_f16_e64 v0, |s0| ; CI-NEXT: s_lshr_b32 s0, s0, 16 @@ -268,6 +295,9 @@ define amdgpu_kernel void @fabs_fold_f16(ptr addrspace(1) %out, half %in0, half ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s3 @@ -325,6 +355,9 @@ define amdgpu_kernel void @v_fabs_v2f16(ptr addrspace(1) %out, ptr addrspace(1) ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 ; CI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_add_i32_e32 v0, vcc, s0, v0 @@ -339,6 +372,9 @@ define amdgpu_kernel void @v_fabs_v2f16(ptr addrspace(1) %out, ptr addrspace(1) ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x8 ; VI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -386,6 +422,9 @@ define amdgpu_kernel void @fabs_free_v2f16(ptr addrspace(1) %out, i32 %in) #0 { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_and_b32 s2, s2, 0x7fff7fff ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -398,6 +437,9 @@ define amdgpu_kernel void @fabs_free_v2f16(ptr addrspace(1) %out, i32 %in) #0 { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_and_b32 s2, s2, 0x7fff7fff ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -441,6 +483,9 @@ define amdgpu_kernel void @v_fabs_fold_self_v2f16(ptr addrspace(1) %out, ptr add ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v0 @@ -467,6 +512,9 @@ define amdgpu_kernel void @v_fabs_fold_self_v2f16(ptr addrspace(1) %out, ptr add ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -523,9 +571,12 @@ define amdgpu_kernel void @v_fabs_fold_v2f16(ptr addrspace(1) %out, ptr addrspac ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v0 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dword v0, v[0:1] ; CI-NEXT: s_lshr_b32 s2, s4, 16 @@ -551,9 +602,12 @@ define amdgpu_kernel void @v_fabs_fold_v2f16(ptr addrspace(1) %out, ptr addrspac ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dword v2, v[0:1] ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -610,6 +664,9 @@ define amdgpu_kernel void @v_extract_fabs_fold_v2f16(ptr addrspace(1) %in) #0 { ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_add_i32_e32 v0, vcc, s0, v0 @@ -633,6 +690,9 @@ define amdgpu_kernel void @v_extract_fabs_fold_v2f16(ptr addrspace(1) %in) #0 { ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -718,6 +778,9 @@ define amdgpu_kernel void @v_extract_fabs_no_fold_v2f16(ptr addrspace(1) %in) #0 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_add_i32_e32 v0, vcc, s0, v0 @@ -736,6 +799,9 @@ define amdgpu_kernel void @v_extract_fabs_no_fold_v2f16(ptr addrspace(1) %in) #0 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_add_u32_e32 v0, vcc, s0, v0 diff --git a/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.global.ll b/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.global.ll index 6496b70b4d697..60334e46a4454 100644 --- a/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.global.ll +++ b/llvm/test/CodeGen/AMDGPU/fast-unaligned-load-store.global.ll @@ -74,6 +74,9 @@ define amdgpu_kernel void @global_store_2xi16_align2(ptr addrspace(1) %p, ptr ad ; GFX7-ALIGNED-LABEL: global_store_2xi16_align2: ; GFX7-ALIGNED: ; %bb.0: ; GFX7-ALIGNED-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 +; GFX7-ALIGNED-NEXT: s_add_i32 s12, s12, s17 +; GFX7-ALIGNED-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-ALIGNED-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v2, 1 ; GFX7-ALIGNED-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v0, s0 @@ -90,6 +93,9 @@ define amdgpu_kernel void @global_store_2xi16_align2(ptr addrspace(1) %p, ptr ad ; GFX7-UNALIGNED-LABEL: global_store_2xi16_align2: ; GFX7-UNALIGNED: ; %bb.0: ; GFX7-UNALIGNED-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 +; GFX7-UNALIGNED-NEXT: s_add_i32 s12, s12, s17 +; GFX7-UNALIGNED-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-UNALIGNED-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-UNALIGNED-NEXT: v_mov_b32_e32 v2, 0x20001 ; GFX7-UNALIGNED-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-UNALIGNED-NEXT: v_mov_b32_e32 v0, s0 @@ -216,8 +222,10 @@ define amdgpu_kernel void @global_store_2xi16_align1(ptr addrspace(1) %p, ptr ad ; GFX7-ALIGNED-LABEL: global_store_2xi16_align1: ; GFX7-ALIGNED: ; %bb.0: ; GFX7-ALIGNED-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 +; GFX7-ALIGNED-NEXT: s_add_i32 s12, s12, s17 +; GFX7-ALIGNED-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-ALIGNED-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v2, 1 -; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v3, 0 ; GFX7-ALIGNED-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-ALIGNED-NEXT: s_add_u32 s2, s0, 2 ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v0, s0 @@ -227,6 +235,7 @@ define amdgpu_kernel void @global_store_2xi16_align1(ptr addrspace(1) %p, ptr ad ; GFX7-ALIGNED-NEXT: flat_store_byte v[0:1], v2 ; GFX7-ALIGNED-NEXT: s_addc_u32 s5, s1, 0 ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v0, s4 +; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v3, 0 ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v1, s5 ; GFX7-ALIGNED-NEXT: s_add_u32 s0, s0, 3 ; GFX7-ALIGNED-NEXT: flat_store_byte v[0:1], v3 @@ -243,6 +252,9 @@ define amdgpu_kernel void @global_store_2xi16_align1(ptr addrspace(1) %p, ptr ad ; GFX7-UNALIGNED-LABEL: global_store_2xi16_align1: ; GFX7-UNALIGNED: ; %bb.0: ; GFX7-UNALIGNED-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 +; GFX7-UNALIGNED-NEXT: s_add_i32 s12, s12, s17 +; GFX7-UNALIGNED-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-UNALIGNED-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-UNALIGNED-NEXT: v_mov_b32_e32 v2, 0x20001 ; GFX7-UNALIGNED-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-UNALIGNED-NEXT: v_mov_b32_e32 v0, s0 @@ -351,6 +363,9 @@ define amdgpu_kernel void @global_store_2xi16_align4(ptr addrspace(1) %p, ptr ad ; GFX7-ALIGNED-LABEL: global_store_2xi16_align4: ; GFX7-ALIGNED: ; %bb.0: ; GFX7-ALIGNED-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 +; GFX7-ALIGNED-NEXT: s_add_i32 s12, s12, s17 +; GFX7-ALIGNED-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-ALIGNED-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v2, 0x20001 ; GFX7-ALIGNED-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-ALIGNED-NEXT: v_mov_b32_e32 v0, s0 @@ -361,6 +376,9 @@ define amdgpu_kernel void @global_store_2xi16_align4(ptr addrspace(1) %p, ptr ad ; GFX7-UNALIGNED-LABEL: global_store_2xi16_align4: ; GFX7-UNALIGNED: ; %bb.0: ; GFX7-UNALIGNED-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 +; GFX7-UNALIGNED-NEXT: s_add_i32 s12, s12, s17 +; GFX7-UNALIGNED-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-UNALIGNED-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-UNALIGNED-NEXT: v_mov_b32_e32 v2, 0x20001 ; GFX7-UNALIGNED-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-UNALIGNED-NEXT: v_mov_b32_e32 v0, s0 diff --git a/llvm/test/CodeGen/AMDGPU/fcanonicalize.ll b/llvm/test/CodeGen/AMDGPU/fcanonicalize.ll index 4e12a30c6f6f4..9919497acea73 100644 --- a/llvm/test/CodeGen/AMDGPU/fcanonicalize.ll +++ b/llvm/test/CodeGen/AMDGPU/fcanonicalize.ll @@ -24,6 +24,9 @@ define amdgpu_kernel void @v_test_canonicalize_var_f32(ptr addrspace(1) %out) #1 ; GFX678-LABEL: v_test_canonicalize_var_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -76,6 +79,9 @@ define amdgpu_kernel void @s_test_canonicalize_var_f32(ptr addrspace(1) %out, fl ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dword s2, s[8:9], 0x2 ; GFX6-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mul_f32_e64 v2, 1.0, s2 ; GFX6-NEXT: v_mov_b32_e32 v0, s0 @@ -87,6 +93,9 @@ define amdgpu_kernel void @s_test_canonicalize_var_f32(ptr addrspace(1) %out, fl ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s2, s[8:9], 0x8 ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mul_f32_e64 v2, 1.0, s2 ; GFX8-NEXT: v_mov_b32_e32 v0, s0 @@ -132,6 +141,9 @@ define amdgpu_kernel void @v_test_canonicalize_fabs_var_f32(ptr addrspace(1) %ou ; GFX678-LABEL: v_test_canonicalize_fabs_var_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -184,6 +196,9 @@ define amdgpu_kernel void @v_test_canonicalize_fneg_fabs_var_f32(ptr addrspace(1 ; GFX678-LABEL: v_test_canonicalize_fneg_fabs_var_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -237,6 +252,9 @@ define amdgpu_kernel void @v_test_canonicalize_fneg_var_f32(ptr addrspace(1) %ou ; GFX678-LABEL: v_test_canonicalize_fneg_var_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -289,6 +307,9 @@ define amdgpu_kernel void @test_fold_canonicalize_undef_f32(ptr addrspace(1) %ou ; GFX678-LABEL: test_fold_canonicalize_undef_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -328,6 +349,9 @@ define amdgpu_kernel void @test_fold_canonicalize_p0_f32(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_p0_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -367,6 +391,9 @@ define amdgpu_kernel void @test_fold_canonicalize_n0_f32(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_n0_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_bfrev_b32_e32 v2, 1 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -409,6 +436,9 @@ define amdgpu_kernel void @test_fold_canonicalize_p1_f32(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_p1_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 1.0 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -449,6 +479,9 @@ define amdgpu_kernel void @test_fold_canonicalize_n1_f32(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_n1_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, -1.0 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -489,6 +522,9 @@ define amdgpu_kernel void @test_fold_canonicalize_literal_f32(ptr addrspace(1) % ; GFX678-LABEL: test_fold_canonicalize_literal_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x41800000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -529,6 +565,9 @@ define amdgpu_kernel void @test_no_denormals_fold_canonicalize_denormal0_f32(ptr ; GFX678-LABEL: test_no_denormals_fold_canonicalize_denormal0_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -568,10 +607,13 @@ define amdgpu_kernel void @test_no_denormals_fold_canonicalize_denormal0_f32_dyn ; GFX678-LABEL: test_no_denormals_fold_canonicalize_denormal0_f32_dynamic: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 ; GFX678-NEXT: s_mov_b32 s2, 0x7fffff -; GFX678-NEXT: v_mul_f32_e64 v2, 1.0, s2 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 +; GFX678-NEXT: v_mul_f32_e64 v2, 1.0, s2 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 ; GFX678-NEXT: flat_store_dword v[0:1], v2 ; GFX678-NEXT: s_endpgm @@ -612,10 +654,13 @@ define amdgpu_kernel void @test_no_denormals_fold_canonicalize_denormal0_f32_dyn ; GFX678-LABEL: test_no_denormals_fold_canonicalize_denormal0_f32_dynamic_out: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 ; GFX678-NEXT: s_mov_b32 s2, 0x7fffff -; GFX678-NEXT: v_mul_f32_e64 v2, 1.0, s2 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 +; GFX678-NEXT: v_mul_f32_e64 v2, 1.0, s2 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 ; GFX678-NEXT: flat_store_dword v[0:1], v2 ; GFX678-NEXT: s_endpgm @@ -656,10 +701,13 @@ define amdgpu_kernel void @test_no_denormals_fold_canonicalize_denormal0_f32_dyn ; GFX678-LABEL: test_no_denormals_fold_canonicalize_denormal0_f32_dynamic_in: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 ; GFX678-NEXT: s_mov_b32 s2, 0x7fffff -; GFX678-NEXT: v_mul_f32_e64 v2, 1.0, s2 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 +; GFX678-NEXT: v_mul_f32_e64 v2, 1.0, s2 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 ; GFX678-NEXT: flat_store_dword v[0:1], v2 ; GFX678-NEXT: s_endpgm @@ -700,6 +748,9 @@ define amdgpu_kernel void @test_denormals_fold_canonicalize_denormal0_f32(ptr ad ; GFX678-LABEL: test_denormals_fold_canonicalize_denormal0_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fffff ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -740,6 +791,9 @@ define amdgpu_kernel void @test_no_denormals_fold_canonicalize_denormal1_f32(ptr ; GFX678-LABEL: test_no_denormals_fold_canonicalize_denormal1_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_bfrev_b32_e32 v2, 1 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -782,6 +836,9 @@ define amdgpu_kernel void @test_denormals_fold_canonicalize_denormal1_f32(ptr ad ; GFX678-LABEL: test_denormals_fold_canonicalize_denormal1_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x807fffff ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -822,6 +879,9 @@ define amdgpu_kernel void @test_fold_canonicalize_qnan_f32(ptr addrspace(1) %out ; GFX678-LABEL: test_fold_canonicalize_qnan_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fc00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -862,6 +922,9 @@ define amdgpu_kernel void @test_fold_canonicalize_qnan_value_neg1_f32(ptr addrsp ; GFX678-LABEL: test_fold_canonicalize_qnan_value_neg1_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fc00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -902,6 +965,9 @@ define amdgpu_kernel void @test_fold_canonicalize_qnan_value_neg2_f32(ptr addrsp ; GFX678-LABEL: test_fold_canonicalize_qnan_value_neg2_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fc00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -942,6 +1008,9 @@ define amdgpu_kernel void @test_fold_canonicalize_snan0_value_f32(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan0_value_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fc00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -982,6 +1051,9 @@ define amdgpu_kernel void @test_fold_canonicalize_snan1_value_f32(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan1_value_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fc00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -1022,6 +1094,9 @@ define amdgpu_kernel void @test_fold_canonicalize_snan2_value_f32(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan2_value_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fc00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -1062,6 +1137,9 @@ define amdgpu_kernel void @test_fold_canonicalize_snan3_value_f32(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan3_value_f32: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v2, 0x7fc00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 @@ -1102,6 +1180,9 @@ define amdgpu_kernel void @v_test_canonicalize_var_f64(ptr addrspace(1) %out) #1 ; GFX678-LABEL: v_test_canonicalize_var_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -1153,6 +1234,9 @@ define amdgpu_kernel void @s_test_canonicalize_var_f64(ptr addrspace(1) %out, do ; GFX6-LABEL: s_test_canonicalize_var_f64: ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_max_f64 v[2:3], s[2:3], s[2:3] ; GFX6-NEXT: v_mov_b32_e32 v0, s0 @@ -1163,6 +1247,9 @@ define amdgpu_kernel void @s_test_canonicalize_var_f64(ptr addrspace(1) %out, do ; GFX8-LABEL: s_test_canonicalize_var_f64: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_max_f64 v[0:1], s[2:3], s[2:3] ; GFX8-NEXT: v_mov_b32_e32 v2, s0 @@ -1205,6 +1292,9 @@ define amdgpu_kernel void @v_test_canonicalize_fabs_var_f64(ptr addrspace(1) %ou ; GFX678-LABEL: v_test_canonicalize_fabs_var_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -1257,6 +1347,9 @@ define amdgpu_kernel void @v_test_canonicalize_fneg_fabs_var_f64(ptr addrspace(1 ; GFX678-LABEL: v_test_canonicalize_fneg_fabs_var_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -1310,6 +1403,9 @@ define amdgpu_kernel void @v_test_canonicalize_fneg_var_f64(ptr addrspace(1) %ou ; GFX678-LABEL: v_test_canonicalize_fneg_var_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v0, s0 ; GFX678-NEXT: v_mov_b32_e32 v1, s1 @@ -1362,10 +1458,13 @@ define amdgpu_kernel void @test_fold_canonicalize_p0_f64(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_p0_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, v0 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, v0 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1407,10 +1506,13 @@ define amdgpu_kernel void @test_fold_canonicalize_n0_f64(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_n0_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_bfrev_b32_e32 v1, 1 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_bfrev_b32_e32 v1, 1 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1450,10 +1552,13 @@ define amdgpu_kernel void @test_fold_canonicalize_p1_f64(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_p1_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x3ff00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x3ff00000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1491,10 +1596,13 @@ define amdgpu_kernel void @test_fold_canonicalize_n1_f64(ptr addrspace(1) %out) ; GFX678-LABEL: test_fold_canonicalize_n1_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0xbff00000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0xbff00000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1532,10 +1640,13 @@ define amdgpu_kernel void @test_fold_canonicalize_literal_f64(ptr addrspace(1) % ; GFX678-LABEL: test_fold_canonicalize_literal_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x40300000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x40300000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1573,10 +1684,13 @@ define amdgpu_kernel void @test_no_denormals_fold_canonicalize_denormal0_f64(ptr ; GFX678-LABEL: test_no_denormals_fold_canonicalize_denormal0_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, v0 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, v0 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1618,10 +1732,13 @@ define amdgpu_kernel void @test_denormals_fold_canonicalize_denormal0_f64(ptr ad ; GFX678-LABEL: test_denormals_fold_canonicalize_denormal0_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, -1 -; GFX678-NEXT: v_mov_b32_e32 v1, 0xfffff ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0xfffff ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1662,10 +1779,13 @@ define amdgpu_kernel void @test_no_denormals_fold_canonicalize_denormal1_f64(ptr ; GFX678-LABEL: test_no_denormals_fold_canonicalize_denormal1_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_bfrev_b32_e32 v1, 1 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_bfrev_b32_e32 v1, 1 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1705,10 +1825,13 @@ define amdgpu_kernel void @test_denormals_fold_canonicalize_denormal1_f64(ptr ad ; GFX678-LABEL: test_denormals_fold_canonicalize_denormal1_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, -1 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x800fffff ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x800fffff ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1749,10 +1872,13 @@ define amdgpu_kernel void @test_fold_canonicalize_qnan_f64(ptr addrspace(1) %out ; GFX678-LABEL: test_fold_canonicalize_qnan_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1790,10 +1916,13 @@ define amdgpu_kernel void @test_fold_canonicalize_qnan_value_neg1_f64(ptr addrsp ; GFX678-LABEL: test_fold_canonicalize_qnan_value_neg1_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1831,10 +1960,13 @@ define amdgpu_kernel void @test_fold_canonicalize_qnan_value_neg2_f64(ptr addrsp ; GFX678-LABEL: test_fold_canonicalize_qnan_value_neg2_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1872,10 +2004,13 @@ define amdgpu_kernel void @test_fold_canonicalize_snan0_value_f64(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan0_value_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1913,10 +2048,13 @@ define amdgpu_kernel void @test_fold_canonicalize_snan1_value_f64(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan1_value_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1954,10 +2092,13 @@ define amdgpu_kernel void @test_fold_canonicalize_snan2_value_f64(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan2_value_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -1995,10 +2136,13 @@ define amdgpu_kernel void @test_fold_canonicalize_snan3_value_f64(ptr addrspace( ; GFX678-LABEL: test_fold_canonicalize_snan3_value_f64: ; GFX678: ; %bb.0: ; GFX678-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX678-NEXT: s_add_i32 s12, s12, s17 +; GFX678-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX678-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX678-NEXT: v_mov_b32_e32 v0, 0 -; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: s_waitcnt lgkmcnt(0) ; GFX678-NEXT: v_mov_b32_e32 v3, s1 +; GFX678-NEXT: v_mov_b32_e32 v1, 0x7ff80000 ; GFX678-NEXT: v_mov_b32_e32 v2, s0 ; GFX678-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; GFX678-NEXT: s_endpgm @@ -2037,6 +2181,9 @@ define amdgpu_kernel void @test_canonicalize_value_f64_flush(ptr addrspace(1) %a ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2054,6 +2201,9 @@ define amdgpu_kernel void @test_canonicalize_value_f64_flush(ptr addrspace(1) %a ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2117,6 +2267,9 @@ define amdgpu_kernel void @test_canonicalize_value_f32_flush(ptr addrspace(1) %a ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2134,6 +2287,9 @@ define amdgpu_kernel void @test_canonicalize_value_f32_flush(ptr addrspace(1) %a ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2197,6 +2353,9 @@ define amdgpu_kernel void @test_canonicalize_value_f16_flush(ptr addrspace(1) %a ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 1, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2215,6 +2374,9 @@ define amdgpu_kernel void @test_canonicalize_value_f16_flush(ptr addrspace(1) %a ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 1, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2279,6 +2441,9 @@ define amdgpu_kernel void @test_canonicalize_value_v2f16_flush(ptr addrspace(1) ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2302,6 +2467,9 @@ define amdgpu_kernel void @test_canonicalize_value_v2f16_flush(ptr addrspace(1) ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2368,6 +2536,9 @@ define amdgpu_kernel void @test_canonicalize_value_f64_denorm(ptr addrspace(1) % ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2385,6 +2556,9 @@ define amdgpu_kernel void @test_canonicalize_value_f64_denorm(ptr addrspace(1) % ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2448,6 +2622,9 @@ define amdgpu_kernel void @test_canonicalize_value_f32_denorm(ptr addrspace(1) % ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2465,6 +2642,9 @@ define amdgpu_kernel void @test_canonicalize_value_f32_denorm(ptr addrspace(1) % ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2529,6 +2709,9 @@ define amdgpu_kernel void @test_canonicalize_value_f16_denorm(ptr addrspace(1) % ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 1, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2547,6 +2730,9 @@ define amdgpu_kernel void @test_canonicalize_value_f16_denorm(ptr addrspace(1) % ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 1, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2612,6 +2798,9 @@ define amdgpu_kernel void @test_canonicalize_value_v2f16_denorm(ptr addrspace(1) ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -2635,6 +2824,9 @@ define amdgpu_kernel void @test_canonicalize_value_v2f16_denorm(ptr addrspace(1) ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -2700,6 +2892,9 @@ define amdgpu_kernel void @v_test_canonicalize_var_v2f64(ptr addrspace(1) %out) ; GFX6: ; %bb.0: ; GFX6-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX6-NEXT: v_lshlrev_b32_e32 v0, 4, v0 +; GFX6-NEXT: s_add_i32 s12, s12, s17 +; GFX6-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX6-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX6-NEXT: s_waitcnt lgkmcnt(0) ; GFX6-NEXT: v_mov_b32_e32 v1, s1 ; GFX6-NEXT: v_add_i32_e32 v0, vcc, s0, v0 @@ -2717,6 +2912,9 @@ define amdgpu_kernel void @v_test_canonicalize_var_v2f64(ptr addrspace(1) %out) ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 4, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 diff --git a/llvm/test/CodeGen/AMDGPU/flat-for-global-subtarget-feature.ll b/llvm/test/CodeGen/AMDGPU/flat-for-global-subtarget-feature.ll index fee6540f43c64..fc316b736d5f1 100644 --- a/llvm/test/CodeGen/AMDGPU/flat-for-global-subtarget-feature.ll +++ b/llvm/test/CodeGen/AMDGPU/flat-for-global-subtarget-feature.ll @@ -22,12 +22,14 @@ ; NOHSA-DEFAULT: buffer_store_dword ; NOHSA-NODEFAULT: flat_store_dword ; NOHSA-NOADDR64: flat_store_dword -define amdgpu_kernel void @test(ptr addrspace(1) %out) { +define amdgpu_kernel void @test(ptr addrspace(1) %out) #0 { entry: store i32 0, ptr addrspace(1) %out ret void } +; ALL-LABEL: {{^}}test_addr64: + ; HSA-DEFAULT: flat_store_dword ; HSA-NODEFAULT: buffer_store_dword ; HSA-NOADDR64: flat_store_dword @@ -35,7 +37,7 @@ entry: ; NOHSA-DEFAULT: buffer_store_dword ; NOHSA-NODEFAULT: flat_store_dword ; NOHSA-NOADDR64: flat_store_dword -define amdgpu_kernel void @test_addr64(ptr addrspace(1) %out) { +define amdgpu_kernel void @test_addr64(ptr addrspace(1) %out) #0 { entry: %out.addr = alloca ptr addrspace(1), align 4, addrspace(5) @@ -51,5 +53,7 @@ entry: ret void } +attributes #0 = { "amdgpu-no-flat-scratch-init" } + !llvm.module.flags = !{!0} !0 = !{i32 1, !"amdhsa_code_object_version", i32 400} diff --git a/llvm/test/CodeGen/AMDGPU/flat-scratch-reg.ll b/llvm/test/CodeGen/AMDGPU/flat-scratch-reg.ll index 45223a24e021a..a59382ba20dc5 100644 --- a/llvm/test/CodeGen/AMDGPU/flat-scratch-reg.ll +++ b/llvm/test/CodeGen/AMDGPU/flat-scratch-reg.ll @@ -8,28 +8,34 @@ ; RUN: llc < %s -mtriple=amdgcn -mcpu=stoney -mattr=+xnack | FileCheck -check-prefix=VI-XNACK -check-prefix=GCN %s ; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri | FileCheck -check-prefixes=GCN %s -; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=-xnack | FileCheck -check-prefixes=VI-NOXNACK,HSA-VI-NOXNACK,GCN %s -; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=+xnack | FileCheck -check-prefixes=VI-XNACK,HSA-VI-XNACK,GCN %s +; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=-xnack | FileCheck -check-prefixes=HSA-VI-NOXNACK,GCN %s +; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo -mattr=+xnack | FileCheck -check-prefixes=HSA-VI-XNACK,GCN %s ; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx900 -mattr=+architected-flat-scratch | FileCheck -check-prefixes=GCN %s -; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx900 -mattr=+architected-flat-scratch,-xnack | FileCheck -check-prefixes=HSA-VI-NOXNACK,GFX9-ARCH-FLAT,GCN %s -; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx900 -mattr=+architected-flat-scratch,+xnack | FileCheck -check-prefixes=HSA-VI-XNACK,GFX9-ARCH-FLAT,GCN %s +; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx900 -mattr=+architected-flat-scratch,-xnack | FileCheck -check-prefixes=GFX9-ARCH-FLAT-NOXNACK,GCN %s +; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx900 -mattr=+architected-flat-scratch,+xnack | FileCheck -check-prefixes=GFX9-ARCH-FLAT-XNACK,GCN %s ; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+architected-flat-scratch | FileCheck -check-prefixes=GCN %s -; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+architected-flat-scratch,-xnack | FileCheck -check-prefixes=HSA-VI-NOXNACK,GFX10-ARCH-FLAT,GCN %s -; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+architected-flat-scratch,+xnack | FileCheck -check-prefixes=HSA-VI-XNACK,GFX10-ARCH-FLAT,GCN %s +; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+architected-flat-scratch,-xnack | FileCheck -check-prefixes=GFX10-ARCH-FLAT-NOXNACK,GCN %s +; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+architected-flat-scratch,+xnack | FileCheck -check-prefixes=GFX10-ARCH-FLAT-XNACK,GCN %s ; GCN-LABEL: {{^}}no_vcc_no_flat: ; NOT-HSA-CI: .amdhsa_reserve_xnack_mask ; HSA-VI-NOXNACK: .amdhsa_reserve_xnack_mask 0 ; HSA-VI-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX9-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX10-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 ; CI: ; TotalNumSgprs: 8 ; VI-NOXNACK: ; TotalNumSgprs: 8 +; HSA-VI-NOXNACK: ; TotalNumSgprs: 8 ; VI-XNACK: ; TotalNumSgprs: 12 -; GFX9-ARCH-FLAT: ; TotalNumSgprs: 14 -; GFX10-ARCH-FLAT: ; TotalNumSgprs: 8 +; HSA-VI-XNACK: ; TotalNumSgprs: 12 +; GFX9-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 14 +; GFX9-ARCH-FLAT-XNACK: ; TotalNumSgprs: 14 +; GFX10-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 8 +; GFX10-ARCH-FLAT-XNACK: ; TotalNumSgprs: 8 define amdgpu_kernel void @no_vcc_no_flat() { entry: call void asm sideeffect "", "~{s7}"() @@ -41,12 +47,18 @@ entry: ; NOT-HSA-CI: .amdhsa_reserve_xnack_mask ; HSA-VI-NOXNACK: .amdhsa_reserve_xnack_mask 0 ; HSA-VI-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX9-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX10-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 ; CI: ; TotalNumSgprs: 10 ; VI-NOXNACK: ; TotalNumSgprs: 10 +; HSA-VI-NOXNACK: ; TotalNumSgprs: 10 ; VI-XNACK: ; TotalNumSgprs: 12 -; GFX9-ARCH-FLAT: ; TotalNumSgprs: 14 -; GFX10-ARCH-FLAT: ; TotalNumSgprs: 10 +; HSA-VI-XNACK: ; TotalNumSgprs: 12 +; GFX9-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 14 +; GFX9-ARCH-FLAT-XNACK: ; TotalNumSgprs: 14 +; GFX10-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 10 +; GFX10-ARCH-FLAT-XNACK: ; TotalNumSgprs: 10 define amdgpu_kernel void @vcc_no_flat() { entry: call void asm sideeffect "", "~{s7},~{vcc}"() @@ -58,12 +70,18 @@ entry: ; NOT-HSA-CI: .amdhsa_reserve_xnack_mask ; HSA-VI-NOXNACK: .amdhsa_reserve_xnack_mask 0 ; HSA-VI-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX9-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX10-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 ; CI: ; TotalNumSgprs: 12 ; VI-NOXNACK: ; TotalNumSgprs: 14 +; HSA-VI-NOXNACK: ; TotalNumSgprs: 24 ; VI-XNACK: ; TotalNumSgprs: 14 -; GFX9-ARCH-FLAT: ; TotalNumSgprs: 14 -; GFX10-ARCH-FLAT: ; TotalNumSgprs: 8 +; HSA-VI-XNACK: ; TotalNumSgprs: 24 +; GFX9-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 14 +; GFX9-ARCH-FLAT-XNACK: ; TotalNumSgprs: 14 +; GFX10-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 8 +; GFX10-ARCH-FLAT-XNACK: ; TotalNumSgprs: 8 define amdgpu_kernel void @no_vcc_flat() { entry: call void asm sideeffect "", "~{s7},~{flat_scratch}"() @@ -75,12 +93,18 @@ entry: ; NOT-HSA-CI: .amdhsa_reserve_xnack_mask ; HSA-VI-NOXNACK: .amdhsa_reserve_xnack_mask 0 ; HSA-VI-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX9-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX10-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 ; CI: ; TotalNumSgprs: 12 ; VI-NOXNACK: ; TotalNumSgprs: 14 +; HSA-VI-NOXNACK: ; TotalNumSgprs: 24 ; VI-XNACK: ; TotalNumSgprs: 14 -; GFX9-ARCH-FLAT: ; TotalNumSgprs: 14 -; GFX10-ARCH-FLAT: ; TotalNumSgprs: 10 +; HSA-VI-XNACK: ; TotalNumSgprs: 24 +; GFX9-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 14 +; GFX9-ARCH-FLAT-XNACK: ; TotalNumSgprs: 14 +; GFX10-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 10 +; GFX10-ARCH-FLAT-XNACK: ; TotalNumSgprs: 10 define amdgpu_kernel void @vcc_flat() { entry: call void asm sideeffect "", "~{s7},~{vcc},~{flat_scratch}"() @@ -95,12 +119,18 @@ entry: ; NOT-HSA-CI: .amdhsa_reserve_xnack_mask ; HSA-VI-NOXNACK: .amdhsa_reserve_xnack_mask 0 ; HSA-VI-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX9-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX10-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 ; CI: NumSgprs: 4 ; VI-NOXNACK: NumSgprs: 6 +; HSA-VI-NOXNACK: NumSgprs: 24 ; VI-XNACK: NumSgprs: 6 -; GFX9-ARCH-FLAT: ; TotalNumSgprs: 6 -; GFX10-ARCH-FLAT: ; TotalNumSgprs: 0 +; HSA-VI-XNACK: NumSgprs: 24 +; GFX9-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 6 +; GFX9-ARCH-FLAT-XNACK: ; TotalNumSgprs: 6 +; GFX10-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 0 +; GFX10-ARCH-FLAT-XNACK: ; TotalNumSgprs: 0 define amdgpu_kernel void @use_flat_scr() #0 { entry: call void asm sideeffect "; clobber ", "~{flat_scratch}"() @@ -115,9 +145,13 @@ entry: ; CI: NumSgprs: 4 ; VI-NOXNACK: NumSgprs: 6 +; HSA-VI-NOXNACK: NumSgprs: 24 ; VI-XNACK: NumSgprs: 6 -; GFX9-ARCH-FLAT: ; TotalNumSgprs: 6 -; GFX10-ARCH-FLAT: ; TotalNumSgprs: 0 +; HSA-VI-XNACK: NumSgprs: 24 +; GFX9-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 6 +; GFX9-ARCH-FLAT-XNACK: ; TotalNumSgprs: 6 +; GFX10-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 0 +; GFX10-ARCH-FLAT-XNACK: ; TotalNumSgprs: 0 define amdgpu_kernel void @use_flat_scr_lo() #0 { entry: call void asm sideeffect "; clobber ", "~{flat_scratch_lo}"() @@ -129,12 +163,18 @@ entry: ; NOT-HSA-CI: .amdhsa_reserve_xnack_mask ; HSA-VI-NOXNACK: .amdhsa_reserve_xnack_mask 0 ; HSA-VI-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX9-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 +; GFX10-ARCH-FLAT-XNACK: .amdhsa_reserve_xnack_mask 1 ; CI: NumSgprs: 4 ; VI-NOXNACK: NumSgprs: 6 +; HSA-VI-NOXNACK: NumSgprs: 24 ; VI-XNACK: NumSgprs: 6 -; GFX9-ARCH-FLAT: ; TotalNumSgprs: 6 -; GFX10-ARCH-FLAT: ; TotalNumSgprs: 0 +; HSA-VI-XNACK: NumSgprs: 24 +; GFX9-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 6 +; GFX9-ARCH-FLAT-XNACK: ; TotalNumSgprs: 6 +; GFX10-ARCH-FLAT-NOXNACK: ; TotalNumSgprs: 0 +; GFX10-ARCH-FLAT-XNACK: ; TotalNumSgprs: 0 define amdgpu_kernel void @use_flat_scr_hi() #0 { entry: call void asm sideeffect "; clobber ", "~{flat_scratch_hi}"() diff --git a/llvm/test/CodeGen/AMDGPU/fmul-2-combine-multi-use.ll b/llvm/test/CodeGen/AMDGPU/fmul-2-combine-multi-use.ll index 64be9cb72a6ee..fb2448fb80744 100644 --- a/llvm/test/CodeGen/AMDGPU/fmul-2-combine-multi-use.ll +++ b/llvm/test/CodeGen/AMDGPU/fmul-2-combine-multi-use.ll @@ -16,6 +16,9 @@ define amdgpu_kernel void @multiple_fadd_use_test_f32(ptr addrspace(1) %out, flo ; VI-LABEL: multiple_fadd_use_test_f32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_add_f32_e64 v0, s3, -1.0 ; VI-NEXT: v_add_f32_e64 v1, s2, -1.0 @@ -80,8 +83,11 @@ define amdgpu_kernel void @multiple_use_fadd_fmac_f32(ptr addrspace(1) %out, flo ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x8 ; VI-NEXT: s_load_dword s3, s[8:9], 0x2c +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_add_u32 s2, s0, 4 ; VI-NEXT: v_add_f32_e64 v2, s4, s4 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -139,6 +145,9 @@ define amdgpu_kernel void @multiple_use_fadd_fmad_f32(ptr addrspace(1) %out, flo ; VI-LABEL: multiple_use_fadd_fmad_f32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: s_add_u32 s4, s0, 4 @@ -194,6 +203,9 @@ define amdgpu_kernel void @multiple_use_fadd_multi_fmad_f32(ptr addrspace(1) %ou ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x8 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s6, s4, 4 ; VI-NEXT: v_mov_b32_e32 v0, s1 @@ -255,6 +267,9 @@ define amdgpu_kernel void @fmul_x2_xn2_f32(ptr addrspace(1) %out, float %x, floa ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mul_f32_e64 v0, s2, -4.0 ; VI-NEXT: v_mul_f32_e32 v2, s2, v0 @@ -303,10 +318,13 @@ define amdgpu_kernel void @fmul_x2_xn3_f32(ptr addrspace(1) %out, float %x, floa ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, 0xc0c00000 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mul_f32_e32 v0, s2, v0 ; VI-NEXT: v_mul_f32_e32 v2, s2, v0 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: s_waitcnt vmcnt(0) @@ -350,6 +368,9 @@ define amdgpu_kernel void @multiple_fadd_use_test_f16(ptr addrspace(1) %out, i16 ; VI-DENORM: ; %bb.0: ; VI-DENORM-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-DENORM-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-DENORM-NEXT: s_add_i32 s12, s12, s17 +; VI-DENORM-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-DENORM-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-DENORM-NEXT: s_waitcnt lgkmcnt(0) ; VI-DENORM-NEXT: s_lshr_b32 s3, s2, 16 ; VI-DENORM-NEXT: v_add_f16_e64 v0, s2, -1.0 @@ -368,6 +389,9 @@ define amdgpu_kernel void @multiple_fadd_use_test_f16(ptr addrspace(1) %out, i16 ; VI-FLUSH: ; %bb.0: ; VI-FLUSH-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-FLUSH-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-FLUSH-NEXT: s_add_i32 s12, s12, s17 +; VI-FLUSH-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-FLUSH-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0) ; VI-FLUSH-NEXT: s_lshr_b32 s3, s2, 16 ; VI-FLUSH-NEXT: v_add_f16_e64 v0, s2, -1.0 @@ -482,6 +506,9 @@ define amdgpu_kernel void @multiple_use_fadd_fmac_f16(ptr addrspace(1) %out, i16 ; VI-DENORM: ; %bb.0: ; VI-DENORM-NEXT: s_load_dword s4, s[8:9], 0x8 ; VI-DENORM-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-DENORM-NEXT: s_add_i32 s12, s12, s17 +; VI-DENORM-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-DENORM-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-DENORM-NEXT: s_waitcnt lgkmcnt(0) ; VI-DENORM-NEXT: s_lshr_b32 s3, s4, 16 ; VI-DENORM-NEXT: v_mov_b32_e32 v0, s3 @@ -503,6 +530,9 @@ define amdgpu_kernel void @multiple_use_fadd_fmac_f16(ptr addrspace(1) %out, i16 ; VI-FLUSH: ; %bb.0: ; VI-FLUSH-NEXT: s_load_dword s4, s[8:9], 0x8 ; VI-FLUSH-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-FLUSH-NEXT: s_add_i32 s12, s12, s17 +; VI-FLUSH-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-FLUSH-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0) ; VI-FLUSH-NEXT: s_lshr_b32 s3, s4, 16 ; VI-FLUSH-NEXT: v_mov_b32_e32 v0, s0 @@ -599,6 +629,9 @@ define amdgpu_kernel void @multiple_use_fadd_fmad_f16(ptr addrspace(1) %out, i16 ; VI-DENORM: ; %bb.0: ; VI-DENORM-NEXT: s_load_dword s4, s[8:9], 0x8 ; VI-DENORM-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-DENORM-NEXT: s_add_i32 s12, s12, s17 +; VI-DENORM-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-DENORM-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-DENORM-NEXT: s_waitcnt lgkmcnt(0) ; VI-DENORM-NEXT: s_lshr_b32 s3, s4, 16 ; VI-DENORM-NEXT: v_mov_b32_e32 v0, s3 @@ -620,6 +653,9 @@ define amdgpu_kernel void @multiple_use_fadd_fmad_f16(ptr addrspace(1) %out, i16 ; VI-FLUSH: ; %bb.0: ; VI-FLUSH-NEXT: s_load_dword s4, s[8:9], 0x8 ; VI-FLUSH-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-FLUSH-NEXT: s_add_i32 s12, s12, s17 +; VI-FLUSH-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-FLUSH-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0) ; VI-FLUSH-NEXT: s_lshr_b32 s3, s4, 16 ; VI-FLUSH-NEXT: v_mov_b32_e32 v0, s3 @@ -718,6 +754,8 @@ define amdgpu_kernel void @multiple_use_fadd_multi_fmad_f16(ptr addrspace(1) %ou ; VI-DENORM-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x8 ; VI-DENORM-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x0 ; VI-DENORM-NEXT: s_load_dword s6, s[8:9], 0x8 +; VI-DENORM-NEXT: s_add_i32 s12, s12, s17 +; VI-DENORM-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-DENORM-NEXT: s_waitcnt lgkmcnt(0) ; VI-DENORM-NEXT: s_lshr_b32 s0, s0, 16 ; VI-DENORM-NEXT: v_mov_b32_e32 v0, s0 @@ -725,6 +763,7 @@ define amdgpu_kernel void @multiple_use_fadd_multi_fmad_f16(ptr addrspace(1) %ou ; VI-DENORM-NEXT: v_mov_b32_e32 v0, s1 ; VI-DENORM-NEXT: v_fma_f16 v3, |s6|, 2.0, v0 ; VI-DENORM-NEXT: v_mov_b32_e32 v0, s2 +; VI-DENORM-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-DENORM-NEXT: s_add_u32 s4, s2, 2 ; VI-DENORM-NEXT: v_mov_b32_e32 v1, s3 ; VI-DENORM-NEXT: s_addc_u32 s5, s3, 0 @@ -741,6 +780,8 @@ define amdgpu_kernel void @multiple_use_fadd_multi_fmad_f16(ptr addrspace(1) %ou ; VI-FLUSH-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x8 ; VI-FLUSH-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x0 ; VI-FLUSH-NEXT: s_load_dword s6, s[8:9], 0x8 +; VI-FLUSH-NEXT: s_add_i32 s12, s12, s17 +; VI-FLUSH-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-FLUSH-NEXT: s_waitcnt lgkmcnt(0) ; VI-FLUSH-NEXT: s_lshr_b32 s0, s0, 16 ; VI-FLUSH-NEXT: v_mov_b32_e32 v0, s0 @@ -748,6 +789,7 @@ define amdgpu_kernel void @multiple_use_fadd_multi_fmad_f16(ptr addrspace(1) %ou ; VI-FLUSH-NEXT: v_mov_b32_e32 v0, s1 ; VI-FLUSH-NEXT: v_mad_f16 v3, |s6|, 2.0, v0 ; VI-FLUSH-NEXT: v_mov_b32_e32 v0, s2 +; VI-FLUSH-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-FLUSH-NEXT: s_add_u32 s4, s2, 2 ; VI-FLUSH-NEXT: v_mov_b32_e32 v1, s3 ; VI-FLUSH-NEXT: s_addc_u32 s5, s3, 0 @@ -847,6 +889,9 @@ define amdgpu_kernel void @fmul_x2_xn2_f16(ptr addrspace(1) %out, i16 zeroext %x ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mul_f16_e64 v0, s2, -4.0 ; VI-NEXT: v_mul_f16_e32 v2, s2, v0 @@ -898,10 +943,13 @@ define amdgpu_kernel void @fmul_x2_xn3_f16(ptr addrspace(1) %out, i16 zeroext %x ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, 0xc600 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mul_f16_e32 v0, s2, v0 ; VI-NEXT: v_mul_f16_e32 v2, s2, v0 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_short v[0:1], v2 ; VI-NEXT: s_waitcnt vmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/fneg-fabs.f16.ll b/llvm/test/CodeGen/AMDGPU/fneg-fabs.f16.ll index 67bec43078803..eca8c2837b0fc 100644 --- a/llvm/test/CodeGen/AMDGPU/fneg-fabs.f16.ll +++ b/llvm/test/CodeGen/AMDGPU/fneg-fabs.f16.ll @@ -9,6 +9,9 @@ define amdgpu_kernel void @fneg_fabs_fadd_f16(ptr addrspace(1) %out, half %x, ha ; CI-LABEL: fneg_fabs_fadd_f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f32_f16_e64 v0, |s0| ; CI-NEXT: s_lshr_b32 s0, s0, 16 @@ -26,6 +29,9 @@ define amdgpu_kernel void @fneg_fabs_fadd_f16(ptr addrspace(1) %out, half %x, ha ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s2 @@ -83,6 +89,9 @@ define amdgpu_kernel void @fneg_fabs_fmul_f16(ptr addrspace(1) %out, half %x, ha ; CI-LABEL: fneg_fabs_fmul_f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_and_b32 s1, s0, 0x7fff ; CI-NEXT: s_lshr_b32 s0, s0, 16 @@ -101,6 +110,9 @@ define amdgpu_kernel void @fneg_fabs_fmul_f16(ptr addrspace(1) %out, half %x, ha ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s2 @@ -162,6 +174,9 @@ define amdgpu_kernel void @fneg_fabs_free_f16(ptr addrspace(1) %out, i16 %in) { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_bitset1_b32 s2, 15 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -174,6 +189,9 @@ define amdgpu_kernel void @fneg_fabs_free_f16(ptr addrspace(1) %out, i16 %in) { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_bitset1_b32 s2, 15 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -229,6 +247,9 @@ define amdgpu_kernel void @fneg_fabs_f16(ptr addrspace(1) %out, half %in) { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_bitset1_b32 s2, 15 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -241,6 +262,9 @@ define amdgpu_kernel void @fneg_fabs_f16(ptr addrspace(1) %out, half %in) { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_bitset1_b32 s2, 15 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -294,6 +318,9 @@ define amdgpu_kernel void @v_fneg_fabs_f16(ptr addrspace(1) %out, ptr addrspace( ; CIVI-LABEL: v_fneg_fabs_f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 @@ -348,6 +375,9 @@ define amdgpu_kernel void @s_fneg_fabs_v2f16_non_bc_src(ptr addrspace(1) %out, < ; CI-LABEL: s_fneg_fabs_v2f16_non_bc_src: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s1, s0, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v1, s1 @@ -370,7 +400,9 @@ define amdgpu_kernel void @s_fneg_fabs_v2f16_non_bc_src(ptr addrspace(1) %out, < ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 0x4000 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v2, s3 @@ -379,6 +411,7 @@ define amdgpu_kernel void @s_fneg_fabs_v2f16_non_bc_src(ptr addrspace(1) %out, < ; VI-NEXT: v_or_b32_e32 v0, v1, v0 ; VI-NEXT: v_or_b32_e32 v2, 0x80008000, v0 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: s_endpgm @@ -421,6 +454,9 @@ define amdgpu_kernel void @s_fneg_fabs_v2f16_bc_src(ptr addrspace(1) %out, <2 x ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_or_b32 s2, s2, 0x80008000 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -433,6 +469,9 @@ define amdgpu_kernel void @s_fneg_fabs_v2f16_bc_src(ptr addrspace(1) %out, <2 x ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_or_b32 s2, s2, 0x80008000 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -473,6 +512,9 @@ define amdgpu_kernel void @fneg_fabs_v4f16(ptr addrspace(1) %out, <4 x half> %in ; CIVI-LABEL: fneg_fabs_v4f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: s_or_b32 s3, s3, 0x80008000 ; CIVI-NEXT: s_or_b32 s2, s2, 0x80008000 @@ -516,6 +558,9 @@ define amdgpu_kernel void @fold_user_fneg_fabs_v2f16(ptr addrspace(1) %out, <2 x ; CI-LABEL: fold_user_fneg_fabs_v2f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s1, s0, 16 ; CI-NEXT: v_cvt_f32_f16_e64 v1, |s1| @@ -537,7 +582,9 @@ define amdgpu_kernel void @fold_user_fneg_fabs_v2f16(ptr addrspace(1) %out, <2 x ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v0, 0xc400 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v2, s3 @@ -545,6 +592,7 @@ define amdgpu_kernel void @fold_user_fneg_fabs_v2f16(ptr addrspace(1) %out, <2 x ; VI-NEXT: v_mul_f16_sdwa v0, |v2|, v0 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD ; VI-NEXT: v_or_b32_e32 v2, v1, v0 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: flat_store_dword v[0:1], v2 ; VI-NEXT: s_endpgm @@ -584,6 +632,9 @@ define amdgpu_kernel void @s_fneg_multi_use_fabs_v2f16(ptr addrspace(1) %out0, p ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: s_and_b32 s0, s4, 0x7fff7fff @@ -601,6 +652,9 @@ define amdgpu_kernel void @s_fneg_multi_use_fabs_v2f16(ptr addrspace(1) %out0, p ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: s_and_b32 s0, s4, 0x7fff7fff @@ -655,6 +709,9 @@ define amdgpu_kernel void @s_fneg_multi_use_fabs_foldable_neg_v2f16(ptr addrspac ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: s_bfe_u32 s0, s4, 0xf0010 @@ -679,7 +736,9 @@ define amdgpu_kernel void @s_fneg_multi_use_fabs_foldable_neg_v2f16(ptr addrspac ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 ; VI-NEXT: v_mov_b32_e32 v5, 0xc400 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: s_lshr_b32 s1, s4, 16 @@ -688,6 +747,7 @@ define amdgpu_kernel void @s_fneg_multi_use_fabs_foldable_neg_v2f16(ptr addrspac ; VI-NEXT: s_and_b32 s0, s4, 0x7fff7fff ; VI-NEXT: v_mul_f16_sdwa v4, |v4|, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD ; VI-NEXT: v_mul_f16_e64 v5, |s4|, -4.0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_or_b32_e32 v4, v5, v4 ; VI-NEXT: v_mov_b32_e32 v5, s0 ; VI-NEXT: v_mov_b32_e32 v2, s2 diff --git a/llvm/test/CodeGen/AMDGPU/fneg-modifier-casting.ll b/llvm/test/CodeGen/AMDGPU/fneg-modifier-casting.ll index 781a2ca3146f5..058c273a65d99 100644 --- a/llvm/test/CodeGen/AMDGPU/fneg-modifier-casting.ll +++ b/llvm/test/CodeGen/AMDGPU/fneg-modifier-casting.ll @@ -1477,6 +1477,8 @@ define amdgpu_kernel void @multiple_uses_fneg_select_f64(double %x, double %y, i ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x4 ; GFX7-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x6 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_bitcmp1_b32 s6, 0 ; GFX7-NEXT: s_cselect_b64 vcc, -1, 0 @@ -1488,6 +1490,7 @@ define amdgpu_kernel void @multiple_uses_fneg_select_f64(double %x, double %y, i ; GFX7-NEXT: s_cselect_b32 s0, s0, s2 ; GFX7-NEXT: v_mov_b32_e32 v1, s1 ; GFX7-NEXT: v_mov_b32_e32 v2, s4 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-NEXT: v_cndmask_b32_e64 v1, v1, -v0, vcc ; GFX7-NEXT: v_mov_b32_e32 v0, s0 ; GFX7-NEXT: v_mov_b32_e32 v3, s5 diff --git a/llvm/test/CodeGen/AMDGPU/fneg.f16.ll b/llvm/test/CodeGen/AMDGPU/fneg.f16.ll index 23e4ba9fd4ed7..98e0b27cd955d 100644 --- a/llvm/test/CodeGen/AMDGPU/fneg.f16.ll +++ b/llvm/test/CodeGen/AMDGPU/fneg.f16.ll @@ -11,6 +11,9 @@ define amdgpu_kernel void @s_fneg_f16(ptr addrspace(1) %out, half %in) #0 { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_xor_b32 s2, s2, 0x8000 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -23,6 +26,9 @@ define amdgpu_kernel void @s_fneg_f16(ptr addrspace(1) %out, half %in) #0 { ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s2, s[8:9], 0x8 ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_xor_b32 s2, s2, 0x8000 ; GFX8-NEXT: v_mov_b32_e32 v0, s0 @@ -78,6 +84,9 @@ define amdgpu_kernel void @v_fneg_f16(ptr addrspace(1) %out, ptr addrspace(1) %i ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 ; CI-NEXT: v_lshlrev_b32_e32 v0, 1, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_add_i32_e32 v0, vcc, s0, v0 @@ -92,6 +101,9 @@ define amdgpu_kernel void @v_fneg_f16(ptr addrspace(1) %out, ptr addrspace(1) %i ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x8 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 1, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -152,6 +164,9 @@ define amdgpu_kernel void @s_fneg_free_f16(ptr addrspace(1) %out, i16 %in) #0 { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_xor_b32 s2, s2, 0x8000 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -164,6 +179,9 @@ define amdgpu_kernel void @s_fneg_free_f16(ptr addrspace(1) %out, i16 %in) #0 { ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s2, s[8:9], 0x8 ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_xor_b32 s2, s2, 0x8000 ; GFX8-NEXT: v_mov_b32_e32 v0, s0 @@ -217,6 +235,9 @@ define amdgpu_kernel void @v_fneg_fold_f16(ptr addrspace(1) %out, ptr addrspace( ; CI-LABEL: v_fneg_fold_f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -234,6 +255,9 @@ define amdgpu_kernel void @v_fneg_fold_f16(ptr addrspace(1) %out, ptr addrspace( ; GFX8-LABEL: v_fneg_fold_f16: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v0, s2 ; GFX8-NEXT: v_mov_b32_e32 v1, s3 @@ -289,6 +313,9 @@ define amdgpu_kernel void @s_fneg_v2f16(ptr addrspace(1) %out, <2 x half> %in) # ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_xor_b32 s2, s2, 0x80008000 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -301,6 +328,9 @@ define amdgpu_kernel void @s_fneg_v2f16(ptr addrspace(1) %out, <2 x half> %in) # ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s2, s[8:9], 0x8 ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_xor_b32 s2, s2, 0x80008000 ; GFX8-NEXT: v_mov_b32_e32 v0, s0 @@ -340,14 +370,17 @@ define amdgpu_kernel void @s_fneg_v2f16_nonload(ptr addrspace(1) %out) #0 { ; CIVI-LABEL: s_fneg_v2f16_nonload: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 ; CIVI-NEXT: ;;#ASMSTART ; CIVI-NEXT: ; def s2 ; CIVI-NEXT: ;;#ASMEND ; CIVI-NEXT: s_xor_b32 s2, s2, 0x80008000 -; CIVI-NEXT: v_mov_b32_e32 v2, s2 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s0 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: v_mov_b32_e32 v1, s1 +; CIVI-NEXT: v_mov_b32_e32 v2, s2 ; CIVI-NEXT: flat_store_dword v[0:1], v2 ; CIVI-NEXT: s_endpgm ; @@ -388,6 +421,9 @@ define amdgpu_kernel void @v_fneg_v2f16(ptr addrspace(1) %out, ptr addrspace(1) ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 ; CI-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_add_i32_e32 v0, vcc, s0, v0 @@ -402,6 +438,9 @@ define amdgpu_kernel void @v_fneg_v2f16(ptr addrspace(1) %out, ptr addrspace(1) ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x8 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -449,6 +488,9 @@ define amdgpu_kernel void @fneg_free_v2f16(ptr addrspace(1) %out, i32 %in) #0 { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_xor_b32 s2, s2, 0x80008000 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -461,6 +503,9 @@ define amdgpu_kernel void @fneg_free_v2f16(ptr addrspace(1) %out, i32 %in) #0 { ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dword s2, s[8:9], 0x8 ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: s_xor_b32 s2, s2, 0x80008000 ; GFX8-NEXT: v_mov_b32_e32 v0, s0 @@ -501,6 +546,9 @@ define amdgpu_kernel void @v_fneg_fold_v2f16(ptr addrspace(1) %out, ptr addrspac ; CI-LABEL: v_fneg_fold_v2f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -527,6 +575,9 @@ define amdgpu_kernel void @v_fneg_fold_v2f16(ptr addrspace(1) %out, ptr addrspac ; GFX8-LABEL: v_fneg_fold_v2f16: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v0, s2 ; GFX8-NEXT: v_mov_b32_e32 v1, s3 @@ -572,6 +623,9 @@ define amdgpu_kernel void @v_extract_fneg_fold_v2f16(ptr addrspace(1) %in) #0 { ; CI-LABEL: v_extract_fneg_fold_v2f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -593,6 +647,9 @@ define amdgpu_kernel void @v_extract_fneg_fold_v2f16(ptr addrspace(1) %in) #0 { ; GFX8-LABEL: v_extract_fneg_fold_v2f16: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v0, s0 ; GFX8-NEXT: v_mov_b32_e32 v1, s1 @@ -672,6 +729,9 @@ define amdgpu_kernel void @v_extract_fneg_no_fold_v2f16(ptr addrspace(1) %in) #0 ; CIVI-LABEL: v_extract_fneg_no_fold_v2f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s0 ; CIVI-NEXT: v_mov_b32_e32 v1, s1 diff --git a/llvm/test/CodeGen/AMDGPU/half.ll b/llvm/test/CodeGen/AMDGPU/half.ll index a2fca33af1046..10573aad38a51 100644 --- a/llvm/test/CodeGen/AMDGPU/half.ll +++ b/llvm/test/CodeGen/AMDGPU/half.ll @@ -10,6 +10,9 @@ define amdgpu_kernel void @load_f16_arg(ptr addrspace(1) %out, half %arg) #0 { ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -21,6 +24,9 @@ define amdgpu_kernel void @load_f16_arg(ptr addrspace(1) %out, half %arg) #0 { ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -46,6 +52,9 @@ define amdgpu_kernel void @load_v2f16_arg(ptr addrspace(1) %out, <2 x half> %arg ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -57,6 +66,9 @@ define amdgpu_kernel void @load_v2f16_arg(ptr addrspace(1) %out, <2 x half> %arg ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -81,6 +93,9 @@ define amdgpu_kernel void @load_v3f16_arg(ptr addrspace(1) %out, <3 x half> %arg ; CIVI-LABEL: load_v3f16_arg: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: s_add_u32 s4, s0, 4 ; CIVI-NEXT: s_addc_u32 s5, s1, 0 @@ -114,6 +129,9 @@ define amdgpu_kernel void @load_v4f16_arg(ptr addrspace(1) %out, <4 x half> %arg ; CIVI-LABEL: load_v4f16_arg: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s0 ; CIVI-NEXT: v_mov_b32_e32 v2, s2 @@ -139,6 +157,9 @@ define amdgpu_kernel void @load_v8f16_arg(ptr addrspace(1) %out, <8 x half> %arg ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v4, s4 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -153,6 +174,9 @@ define amdgpu_kernel void @load_v8f16_arg(ptr addrspace(1) %out, <8 x half> %arg ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v4, s4 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -183,6 +207,9 @@ define amdgpu_kernel void @extload_v2f16_arg(ptr addrspace(1) %out, <2 x half> % ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s3, s2, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v1, s3 @@ -196,6 +223,9 @@ define amdgpu_kernel void @extload_v2f16_arg(ptr addrspace(1) %out, <2 x half> % ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_cvt_f32_f16_e32 v1, s3 @@ -227,6 +257,9 @@ define amdgpu_kernel void @extload_f16_to_f32_arg(ptr addrspace(1) %out, half %a ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f32_f16_e32 v2, s2 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -238,6 +271,9 @@ define amdgpu_kernel void @extload_f16_to_f32_arg(ptr addrspace(1) %out, half %a ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f32_f16_e32 v2, s2 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -265,6 +301,9 @@ define amdgpu_kernel void @extload_v2f16_to_v2f32_arg(ptr addrspace(1) %out, <2 ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s3, s2, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v1, s3 @@ -278,6 +317,9 @@ define amdgpu_kernel void @extload_v2f16_to_v2f32_arg(ptr addrspace(1) %out, <2 ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_cvt_f32_f16_e32 v1, s3 @@ -308,6 +350,9 @@ define amdgpu_kernel void @extload_v3f16_to_v3f32_arg(ptr addrspace(1) %out, <3 ; CI-LABEL: extload_v3f16_to_v3f32_arg: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s4, s2, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v2, s3 @@ -321,6 +366,9 @@ define amdgpu_kernel void @extload_v3f16_to_v3f32_arg(ptr addrspace(1) %out, <3 ; VI-LABEL: extload_v3f16_to_v3f32_arg: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s4, s2, 16 ; VI-NEXT: v_cvt_f32_f16_e32 v0, s2 @@ -351,6 +399,9 @@ define amdgpu_kernel void @extload_v4f16_to_v4f32_arg(ptr addrspace(1) %out, <4 ; CI-LABEL: extload_v4f16_to_v4f32_arg: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s4, s3, 16 ; CI-NEXT: s_lshr_b32 s5, s2, 16 @@ -366,6 +417,9 @@ define amdgpu_kernel void @extload_v4f16_to_v4f32_arg(ptr addrspace(1) %out, <4 ; VI-LABEL: extload_v4f16_to_v4f32_arg: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s4, s3, 16 ; VI-NEXT: s_lshr_b32 s5, s2, 16 @@ -401,6 +455,9 @@ define amdgpu_kernel void @extload_v8f16_to_v8f32_arg(ptr addrspace(1) %out, <8 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x4 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s6, s1, 16 ; CI-NEXT: s_lshr_b32 s7, s0, 16 @@ -429,6 +486,9 @@ define amdgpu_kernel void @extload_v8f16_to_v8f32_arg(ptr addrspace(1) %out, <8 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x10 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s6, s1, 16 ; VI-NEXT: s_lshr_b32 s7, s0, 16 @@ -485,6 +545,9 @@ define amdgpu_kernel void @extload_f16_to_f64_arg(ptr addrspace(1) %out, half %a ; CI-LABEL: extload_f16_to_f64_arg: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f32_f16_e32 v0, s0 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -498,6 +561,9 @@ define amdgpu_kernel void @extload_f16_to_f64_arg(ptr addrspace(1) %out, half %a ; VI-LABEL: extload_f16_to_f64_arg: ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s0, s[8:9], 0x8 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f32_f16_e32 v0, s0 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -529,6 +595,9 @@ define amdgpu_kernel void @extload_v2f16_to_v2f64_arg(ptr addrspace(1) %out, <2 ; CI-LABEL: extload_v2f16_to_v2f64_arg: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s1, s0, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v0, s1 @@ -545,6 +614,9 @@ define amdgpu_kernel void @extload_v2f16_to_v2f64_arg(ptr addrspace(1) %out, <2 ; VI-LABEL: extload_v2f16_to_v2f64_arg: ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s0, s[8:9], 0x8 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s1, s0, 16 ; VI-NEXT: v_cvt_f32_f16_e32 v0, s1 @@ -582,6 +654,9 @@ define amdgpu_kernel void @extload_v3f16_to_v3f64_arg(ptr addrspace(1) %out, <3 ; CI-LABEL: extload_v3f16_to_v3f64_arg: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f32_f16_e32 v0, s3 ; CI-NEXT: s_lshr_b32 s4, s2, 16 @@ -603,6 +678,9 @@ define amdgpu_kernel void @extload_v3f16_to_v3f64_arg(ptr addrspace(1) %out, <3 ; VI-LABEL: extload_v3f16_to_v3f64_arg: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f32_f16_e32 v1, s3 ; VI-NEXT: s_lshr_b32 s4, s2, 16 @@ -648,6 +726,9 @@ define amdgpu_kernel void @extload_v4f16_to_v4f64_arg(ptr addrspace(1) %out, <4 ; CI-LABEL: extload_v4f16_to_v4f64_arg: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s4, s3, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v0, s3 @@ -673,6 +754,9 @@ define amdgpu_kernel void @extload_v4f16_to_v4f64_arg(ptr addrspace(1) %out, <4 ; VI-LABEL: extload_v4f16_to_v4f64_arg: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s5, s3, 16 ; VI-NEXT: v_cvt_f32_f16_e32 v0, s3 @@ -726,6 +810,9 @@ define amdgpu_kernel void @extload_v8f16_to_v8f64_arg(ptr addrspace(1) %out, <8 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x4 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s6, s3, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v0, s6 @@ -773,6 +860,9 @@ define amdgpu_kernel void @extload_v8f16_to_v8f64_arg(ptr addrspace(1) %out, <8 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x10 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s6, s0, 16 ; VI-NEXT: s_lshr_b32 s8, s2, 16 @@ -858,6 +948,9 @@ define amdgpu_kernel void @global_load_store_f16(ptr addrspace(1) %out, ptr addr ; CIVI-LABEL: global_load_store_f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 @@ -886,6 +979,9 @@ define amdgpu_kernel void @global_load_store_v2f16(ptr addrspace(1) %out, ptr ad ; CIVI-LABEL: global_load_store_v2f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 @@ -914,6 +1010,9 @@ define amdgpu_kernel void @global_load_store_v4f16(ptr addrspace(1) %in, ptr add ; CIVI-LABEL: global_load_store_v4f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s0 ; CIVI-NEXT: v_mov_b32_e32 v1, s1 @@ -942,6 +1041,9 @@ define amdgpu_kernel void @global_load_store_v8f16(ptr addrspace(1) %out, ptr ad ; CIVI-LABEL: global_load_store_v8f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 @@ -970,6 +1072,9 @@ define amdgpu_kernel void @global_extload_f16_to_f32(ptr addrspace(1) %out, ptr ; CIVI-LABEL: global_extload_f16_to_f32: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 @@ -1001,6 +1106,9 @@ define amdgpu_kernel void @global_extload_v2f16_to_v2f32(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v2f16_to_v2f32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1017,6 +1125,9 @@ define amdgpu_kernel void @global_extload_v2f16_to_v2f32(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v2f16_to_v2f32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1052,6 +1163,9 @@ define amdgpu_kernel void @global_extload_v3f16_to_v3f32(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v3f16_to_v3f32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1069,6 +1183,9 @@ define amdgpu_kernel void @global_extload_v3f16_to_v3f32(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v3f16_to_v3f32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1106,6 +1223,9 @@ define amdgpu_kernel void @global_extload_v4f16_to_v4f32(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v4f16_to_v4f32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1125,6 +1245,9 @@ define amdgpu_kernel void @global_extload_v4f16_to_v4f32(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v4f16_to_v4f32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1165,6 +1288,9 @@ define amdgpu_kernel void @global_extload_v8f16_to_v8f32(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v8f16_to_v8f32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1195,6 +1321,9 @@ define amdgpu_kernel void @global_extload_v8f16_to_v8f32(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v8f16_to_v8f32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1251,6 +1380,9 @@ define amdgpu_kernel void @global_extload_v16f16_to_v16f32(ptr addrspace(1) %out ; CI-LABEL: global_extload_v16f16_to_v16f32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s4, s2, 16 ; CI-NEXT: v_mov_b32_e32 v5, s3 @@ -1309,6 +1441,9 @@ define amdgpu_kernel void @global_extload_v16f16_to_v16f32(ptr addrspace(1) %out ; VI-LABEL: global_extload_v16f16_to_v16f32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1406,6 +1541,9 @@ define amdgpu_kernel void @global_extload_f16_to_f64(ptr addrspace(1) %out, ptr ; CIVI-LABEL: global_extload_f16_to_f64: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 @@ -1440,6 +1578,9 @@ define amdgpu_kernel void @global_extload_v2f16_to_v2f64(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v2f16_to_v2f64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1458,6 +1599,9 @@ define amdgpu_kernel void @global_extload_v2f16_to_v2f64(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v2f16_to_v2f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1498,6 +1642,9 @@ define amdgpu_kernel void @global_extload_v3f16_to_v3f64(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v3f16_to_v3f64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1523,6 +1670,9 @@ define amdgpu_kernel void @global_extload_v3f16_to_v3f64(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v3f16_to_v3f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1574,6 +1724,9 @@ define amdgpu_kernel void @global_extload_v4f16_to_v4f64(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v4f16_to_v4f64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1602,6 +1755,9 @@ define amdgpu_kernel void @global_extload_v4f16_to_v4f64(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v4f16_to_v4f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1659,6 +1815,9 @@ define amdgpu_kernel void @global_extload_v8f16_to_v8f64(ptr addrspace(1) %out, ; CI-LABEL: global_extload_v8f16_to_v8f64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1707,6 +1866,9 @@ define amdgpu_kernel void @global_extload_v8f16_to_v8f64(ptr addrspace(1) %out, ; VI-LABEL: global_extload_v8f16_to_v8f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -1791,6 +1953,9 @@ define amdgpu_kernel void @global_extload_v16f16_to_v16f64(ptr addrspace(1) %out ; CI-LABEL: global_extload_v16f16_to_v16f64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -1885,6 +2050,9 @@ define amdgpu_kernel void @global_extload_v16f16_to_v16f64(ptr addrspace(1) %out ; VI-LABEL: global_extload_v16f16_to_v16f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2039,6 +2207,9 @@ define amdgpu_kernel void @global_truncstore_f32_to_f16(ptr addrspace(1) %out, p ; CIVI-LABEL: global_truncstore_f32_to_f16: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 @@ -2070,6 +2241,9 @@ define amdgpu_kernel void @global_truncstore_v2f32_to_v2f16(ptr addrspace(1) %ou ; CI-LABEL: global_truncstore_v2f32_to_v2f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2087,6 +2261,9 @@ define amdgpu_kernel void @global_truncstore_v2f32_to_v2f16(ptr addrspace(1) %ou ; VI-LABEL: global_truncstore_v2f32_to_v2f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2123,6 +2300,9 @@ define amdgpu_kernel void @global_truncstore_v3f32_to_v3f16(ptr addrspace(1) %ou ; CI-LABEL: global_truncstore_v3f32_to_v3f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2146,6 +2326,9 @@ define amdgpu_kernel void @global_truncstore_v3f32_to_v3f16(ptr addrspace(1) %ou ; VI-LABEL: global_truncstore_v3f32_to_v3f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2191,6 +2374,9 @@ define amdgpu_kernel void @global_truncstore_v4f32_to_v4f16(ptr addrspace(1) %ou ; CI-LABEL: global_truncstore_v4f32_to_v4f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2212,6 +2398,9 @@ define amdgpu_kernel void @global_truncstore_v4f32_to_v4f16(ptr addrspace(1) %ou ; VI-LABEL: global_truncstore_v4f32_to_v4f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2254,6 +2443,9 @@ define amdgpu_kernel void @global_truncstore_v8f32_to_v8f16(ptr addrspace(1) %ou ; CI-LABEL: global_truncstore_v8f32_to_v8f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2289,6 +2481,9 @@ define amdgpu_kernel void @global_truncstore_v8f32_to_v8f16(ptr addrspace(1) %ou ; VI-LABEL: global_truncstore_v8f32_to_v8f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2352,6 +2547,9 @@ define amdgpu_kernel void @global_truncstore_v16f32_to_v16f16(ptr addrspace(1) % ; CI-LABEL: global_truncstore_v16f32_to_v16f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_add_u32 s4, s2, 32 ; CI-NEXT: s_addc_u32 s5, s3, 0 @@ -2420,6 +2618,9 @@ define amdgpu_kernel void @global_truncstore_v16f32_to_v16f16(ptr addrspace(1) % ; VI-LABEL: global_truncstore_v16f32_to_v16f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_add_u32 s4, s2, 32 ; VI-NEXT: s_addc_u32 s5, s3, 0 @@ -2530,6 +2731,9 @@ define amdgpu_kernel void @fadd_f16(ptr addrspace(1) %out, half %a, half %b) #0 ; CI-LABEL: fadd_f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s0, s[8:9], 0x2 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f32_f16_e32 v0, s0 ; CI-NEXT: s_lshr_b32 s0, s0, 16 @@ -2547,6 +2751,9 @@ define amdgpu_kernel void @fadd_f16(ptr addrspace(1) %out, half %a, half %b) #0 ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s3, s2, 16 ; VI-NEXT: v_mov_b32_e32 v0, s3 @@ -2577,6 +2784,9 @@ define amdgpu_kernel void @fadd_v2f16(ptr addrspace(1) %out, <2 x half> %a, <2 x ; CI-LABEL: fadd_v2f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s4, s2, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v0, s2 @@ -2598,6 +2808,9 @@ define amdgpu_kernel void @fadd_v2f16(ptr addrspace(1) %out, <2 x half> %a, <2 x ; VI-LABEL: fadd_v2f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s4, s3, 16 ; VI-NEXT: s_lshr_b32 s5, s2, 16 @@ -2629,6 +2842,9 @@ define amdgpu_kernel void @fadd_v4f16(ptr addrspace(1) %out, ptr addrspace(1) %i ; CI-LABEL: fadd_v4f16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s2 ; CI-NEXT: v_mov_b32_e32 v1, s3 @@ -2666,6 +2882,9 @@ define amdgpu_kernel void @fadd_v4f16(ptr addrspace(1) %out, ptr addrspace(1) %i ; VI-LABEL: fadd_v4f16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s2 ; VI-NEXT: v_mov_b32_e32 v1, s3 @@ -2706,6 +2925,9 @@ define amdgpu_kernel void @fadd_v8f16(ptr addrspace(1) %out, <8 x half> %a, <8 x ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x4 ; CI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s10, s0, 16 ; CI-NEXT: v_cvt_f32_f16_e32 v4, s0 @@ -2764,6 +2986,9 @@ define amdgpu_kernel void @fadd_v8f16(ptr addrspace(1) %out, <8 x half> %a, <8 x ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x10 ; VI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s10, s7, 16 ; VI-NEXT: s_lshr_b32 s11, s3, 16 @@ -2824,6 +3049,9 @@ define amdgpu_kernel void @test_bitcast_from_half(ptr addrspace(1) %in, ptr addr ; CIVI-LABEL: test_bitcast_from_half: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s0 ; CIVI-NEXT: v_mov_b32_e32 v1, s1 @@ -2853,6 +3081,9 @@ define amdgpu_kernel void @test_bitcast_to_half(ptr addrspace(1) %out, ptr addrs ; CIVI-LABEL: test_bitcast_to_half: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: v_mov_b32_e32 v0, s2 ; CIVI-NEXT: v_mov_b32_e32 v1, s3 diff --git a/llvm/test/CodeGen/AMDGPU/hsa-metadata-kernel-code-props.ll b/llvm/test/CodeGen/AMDGPU/hsa-metadata-kernel-code-props.ll index 8c017fa5ec263..cd89a36fe538b 100644 --- a/llvm/test/CodeGen/AMDGPU/hsa-metadata-kernel-code-props.ll +++ b/llvm/test/CodeGen/AMDGPU/hsa-metadata-kernel-code-props.ll @@ -23,7 +23,7 @@ define amdgpu_kernel void @test( ptr addrspace(1) %r, ptr addrspace(1) %a, - ptr addrspace(1) %b) "amdgpu-no-implicitarg-ptr" { + ptr addrspace(1) %b) "amdgpu-no-implicitarg-ptr" "amdgpu-no-flat-scratch-init" { entry: %a.val = load half, ptr addrspace(1) %a %b.val = load half, ptr addrspace(1) %b @@ -170,7 +170,7 @@ define amdgpu_kernel void @num_spilled_vgprs() #1 { ; CHECK-NEXT: - 1 ; CHECK-NEXT: - 1 -attributes #0 = { "amdgpu-num-sgpr"="20" } +attributes #0 = { "amdgpu-num-sgpr"="20" "amdgpu-no-flat-scratch-init" } attributes #1 = { "amdgpu-num-vgpr"="20" } attributes #2 = { "amdgpu-flat-work-group-size"="1,256" } diff --git a/llvm/test/CodeGen/AMDGPU/hsa.ll b/llvm/test/CodeGen/AMDGPU/hsa.ll index 5a2a976e23846..024593c49dba1 100644 --- a/llvm/test/CodeGen/AMDGPU/hsa.ll +++ b/llvm/test/CodeGen/AMDGPU/hsa.ll @@ -43,7 +43,7 @@ ; ELF: 00E0: 6E616D65 A673696D 706C65BB 2E707269 ; ELF: 00F0: 76617465 5F736567 6D656E74 5F666978 ; ELF: 0100: 65645F73 697A6500 AB2E7367 70725F63 -; ELF: 0110: 6F756E74 06B12E73 6770725F 7370696C +; ELF: 0110: 6F756E74 0EB12E73 6770725F 7370696C ; ELF: 0120: 6C5F636F 756E7400 A72E7379 6D626F6C ; ELF: 0130: A973696D 706C652E 6B64AB2E 76677072 ; ELF: 0140: 5F636F75 6E7403B1 2E766770 725F7370 @@ -59,7 +59,7 @@ ; ELF: 01E0: 73696D70 6C655F6E 6F5F6B65 726E6172 ; ELF: 01F0: 6773BB2E 70726976 6174655F 7365676D ; ELF: 0200: 656E745F 66697865 645F7369 7A6500AB -; ELF: 0210: 2E736770 725F636F 756E7400 B12E7367 +; ELF: 0210: 2E736770 725F636F 756E740C B12E7367 ; ELF: 0220: 70725F73 70696C6C 5F636F75 6E7400A7 ; ELF: 0230: 2E73796D 626F6CB5 73696D70 6C655F6E ; ELF: 0240: 6F5F6B65 726E6172 67732E6B 64AB2E76 diff --git a/llvm/test/CodeGen/AMDGPU/implicit-kernarg-backend-usage.ll b/llvm/test/CodeGen/AMDGPU/implicit-kernarg-backend-usage.ll index b4b6bef7a7ed3..ec80efc5f0362 100644 --- a/llvm/test/CodeGen/AMDGPU/implicit-kernarg-backend-usage.ll +++ b/llvm/test/CodeGen/AMDGPU/implicit-kernarg-backend-usage.ll @@ -7,7 +7,7 @@ ; RUN: sed 's/CODE_OBJECT_VERSION/500/g' %s | llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx906 | FileCheck --check-prefixes=GFX9V5 %s ; RUN: sed 's/CODE_OBJECT_VERSION/600/g' %s | llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx906 | FileCheck --check-prefixes=GFX9V5 %s -define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addrspace(3) %ptr.local) { +define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addrspace(3) %ptr.local) #0 { ; GFX8V4-LABEL: addrspacecast: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -109,7 +109,7 @@ define amdgpu_kernel void @addrspacecast(ptr addrspace(5) %ptr.private, ptr addr ret void } -define amdgpu_kernel void @llvm_amdgcn_is_shared(ptr %ptr) { +define amdgpu_kernel void @llvm_amdgcn_is_shared(ptr %ptr) #0 { ; GFX8V4-LABEL: llvm_amdgcn_is_shared: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_load_dword s0, s[6:7], 0x40 @@ -163,7 +163,7 @@ define amdgpu_kernel void @llvm_amdgcn_is_shared(ptr %ptr) { ret void } -define amdgpu_kernel void @llvm_amdgcn_is_private(ptr %ptr) { +define amdgpu_kernel void @llvm_amdgcn_is_private(ptr %ptr) #0 { ; GFX8V4-LABEL: llvm_amdgcn_is_private: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: s_load_dword s0, s[6:7], 0x44 @@ -283,7 +283,7 @@ define amdgpu_kernel void @llvm_ubsantrap() { unreachable } -define amdgpu_kernel void @llvm_amdgcn_queue_ptr(ptr addrspace(1) %ptr) { +define amdgpu_kernel void @llvm_amdgcn_queue_ptr(ptr addrspace(1) %ptr) #0 { ; GFX8V4-LABEL: llvm_amdgcn_queue_ptr: ; GFX8V4: ; %bb.0: ; GFX8V4-NEXT: v_mov_b32_e32 v0, s6 @@ -389,3 +389,5 @@ declare void @llvm.debugtrap() !llvm.module.flags = !{!0} !0 = !{i32 1, !"amdhsa_code_object_version", i32 CODE_OBJECT_VERSION} + +attributes #0 = { "amdgpu-no-flat-scratch-init" } diff --git a/llvm/test/CodeGen/AMDGPU/inline-asm.i128.ll b/llvm/test/CodeGen/AMDGPU/inline-asm.i128.ll index 9eb966b4e2a94..0ca180ed6e105 100644 --- a/llvm/test/CodeGen/AMDGPU/inline-asm.i128.ll +++ b/llvm/test/CodeGen/AMDGPU/inline-asm.i128.ll @@ -8,16 +8,16 @@ define amdgpu_kernel void @s_input_output_i128() { ; GFX908-LABEL: name: s_input_output_i128 ; GFX908: bb.0 (%ir-block.0): - ; GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 7208970 /* regdef:SGPR_128 */, def %12 - ; GFX908-NEXT: [[COPY:%[0-9]+]]:sgpr_128 = COPY %12 - ; GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 7208969 /* reguse:SGPR_128 */, [[COPY]] + ; GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 7208970 /* regdef:SGPR_128 */, def %13 + ; GFX908-NEXT: [[COPY:%[0-9]+]]:sgpr_128 = COPY %13 + ; GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 7208969 /* reguse:SGPR_128 */, %14 ; GFX908-NEXT: S_ENDPGM 0 ; ; GFX90A-LABEL: name: s_input_output_i128 ; GFX90A: bb.0 (%ir-block.0): - ; GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 7208970 /* regdef:SGPR_128 */, def %10 - ; GFX90A-NEXT: [[COPY:%[0-9]+]]:sgpr_128 = COPY %10 - ; GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 7208969 /* reguse:SGPR_128 */, [[COPY]] + ; GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 7208970 /* regdef:SGPR_128 */, def %11 + ; GFX90A-NEXT: [[COPY:%[0-9]+]]:sgpr_128 = COPY %11 + ; GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 7208969 /* reguse:SGPR_128 */, %12 ; GFX90A-NEXT: S_ENDPGM 0 %val = tail call i128 asm sideeffect "; def $0", "=s"() call void asm sideeffect "; use $0", "s"(i128 %val) @@ -27,16 +27,16 @@ define amdgpu_kernel void @s_input_output_i128() { define amdgpu_kernel void @v_input_output_i128() { ; GFX908-LABEL: name: v_input_output_i128 ; GFX908: bb.0 (%ir-block.0): - ; GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6094858 /* regdef:VReg_128 */, def %12 - ; GFX908-NEXT: [[COPY:%[0-9]+]]:vreg_128 = COPY %12 - ; GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6094857 /* reguse:VReg_128 */, [[COPY]] + ; GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6094858 /* regdef:VReg_128 */, def %13 + ; GFX908-NEXT: [[COPY:%[0-9]+]]:vreg_128 = COPY %13 + ; GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6094857 /* reguse:VReg_128 */, %14 ; GFX908-NEXT: S_ENDPGM 0 ; ; GFX90A-LABEL: name: v_input_output_i128 ; GFX90A: bb.0 (%ir-block.0): - ; GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6422538 /* regdef:VReg_128_Align2 */, def %10 - ; GFX90A-NEXT: [[COPY:%[0-9]+]]:vreg_128_align2 = COPY %10 - ; GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6422537 /* reguse:VReg_128_Align2 */, [[COPY]] + ; GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6422538 /* regdef:VReg_128_Align2 */, def %11 + ; GFX90A-NEXT: [[COPY:%[0-9]+]]:vreg_128_align2 = COPY %11 + ; GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6422537 /* reguse:VReg_128_Align2 */, %12 ; GFX90A-NEXT: S_ENDPGM 0 %val = tail call i128 asm sideeffect "; def $0", "=v"() call void asm sideeffect "; use $0", "v"(i128 %val) @@ -46,16 +46,17 @@ define amdgpu_kernel void @v_input_output_i128() { define amdgpu_kernel void @a_input_output_i128() { ; GFX908-LABEL: name: a_input_output_i128 ; GFX908: bb.0 (%ir-block.0): - ; GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6029322 /* regdef:AReg_128 */, def %12 - ; GFX908-NEXT: [[COPY:%[0-9]+]]:areg_128 = COPY %12 - ; GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6029321 /* reguse:AReg_128 */, [[COPY]] + ; GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6029322 /* regdef:AReg_128 */, def %13 + ; GFX908-NEXT: [[COPY:%[0-9]+]]:areg_128 = COPY %13 + ; GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6029321 /* reguse:AReg_128 */, %14 + ; GFX908-NEXT: S_ENDPGM 0 ; ; GFX90A-LABEL: name: a_input_output_i128 ; GFX90A: bb.0 (%ir-block.0): - ; GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6291466 /* regdef:AReg_128_Align2 */, def %10 - ; GFX90A-NEXT: [[COPY:%[0-9]+]]:areg_128_align2 = COPY %10 - ; GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6291465 /* reguse:AReg_128_Align2 */, [[COPY]] + ; GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6291466 /* regdef:AReg_128_Align2 */, def %11 + ; GFX90A-NEXT: [[COPY:%[0-9]+]]:areg_128_align2 = COPY %11 + ; GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 6291465 /* reguse:AReg_128_Align2 */, %12 ; GFX90A-NEXT: S_ENDPGM 0 %val = call i128 asm sideeffect "; def $0", "=a"() call void asm sideeffect "; use $0", "a"(i128 %val) diff --git a/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2bf16.ll b/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2bf16.ll index 75db7571444bc..b51cb9df8d784 100644 --- a/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2bf16.ll +++ b/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2bf16.ll @@ -22,6 +22,9 @@ define amdgpu_kernel void @s_insertelement_v2bf16_0(ptr addrspace(1) %out, ptr a ; VI-LABEL: s_insertelement_v2bf16_0: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -82,6 +85,9 @@ define amdgpu_kernel void @s_insertelement_v2bf16_1(ptr addrspace(1) %out, ptr a ; VI-LABEL: s_insertelement_v2bf16_1: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -144,6 +150,9 @@ define amdgpu_kernel void @v_insertelement_v2bf16_0(ptr addrspace(1) %out, ptr a ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -216,6 +225,9 @@ define amdgpu_kernel void @v_insertelement_v2bf16_0_inlineimm(ptr addrspace(1) % ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -286,6 +298,9 @@ define amdgpu_kernel void @v_insertelement_v2bf16_1(ptr addrspace(1) %out, ptr a ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -358,6 +373,9 @@ define amdgpu_kernel void @v_insertelement_v2bf16_1_inlineimm(ptr addrspace(1) % ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -435,11 +453,14 @@ define amdgpu_kernel void @v_insertelement_v2bf16_dynamic_vgpr(ptr addrspace(1) ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) -; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: v_mov_b32_e32 v1, s5 ; VI-NEXT: v_add_u32_e32 v0, vcc, s4, v2 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_load_dword v4, v[0:1] ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v3, vcc @@ -531,14 +552,17 @@ define amdgpu_kernel void @v_insertelement_v4bf16_0(ptr addrspace(1) %out, ptr a ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v0, s4, v0, v4 @@ -611,14 +635,17 @@ define amdgpu_kernel void @v_insertelement_v4bf16_1(ptr addrspace(1) %out, ptr a ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v0, v0, s4, v4 @@ -689,14 +716,17 @@ define amdgpu_kernel void @v_insertelement_v4bf16_2(ptr addrspace(1) %out, ptr a ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v1, s4, v1, v4 @@ -769,14 +799,17 @@ define amdgpu_kernel void @v_insertelement_v4bf16_3(ptr addrspace(1) %out, ptr a ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v1, v1, s4, v4 @@ -853,9 +886,12 @@ define amdgpu_kernel void @v_insertelement_v4bf16_dynamic_sgpr(ptr addrspace(1) ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 @@ -948,9 +984,12 @@ define amdgpu_kernel void @v_insertelement_v8bf16_3(ptr addrspace(1) %out, ptr a ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: v_add_u32_e32 v4, vcc, s0, v4 @@ -1065,9 +1104,12 @@ define amdgpu_kernel void @v_insertelement_v8bf16_dynamic(ptr addrspace(1) %out, ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: v_mov_b32_e32 v5, s1 @@ -1245,11 +1287,14 @@ define amdgpu_kernel void @v_insertelement_v16bf16_3(ptr addrspace(1) %out, ptr ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v8, 5, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_add_u32_e32 v4, vcc, 16, v0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v5, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: flat_load_dwordx4 v[4:7], v[4:5] @@ -1417,11 +1462,14 @@ define amdgpu_kernel void @v_insertelement_v16bf16_dynamic(ptr addrspace(1) %out ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v8, 5, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s3 ; VI-NEXT: v_add_u32_e32 v4, vcc, s2, v8 ; VI-NEXT: v_addc_u32_e32 v5, vcc, 0, v0, vcc ; VI-NEXT: v_add_u32_e32 v0, vcc, 16, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v5, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: flat_load_dwordx4 v[4:7], v[4:5] diff --git a/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2i16.ll b/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2i16.ll index 97c97ac8a7ad3..e11900ac0ca68 100644 --- a/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2i16.ll +++ b/llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2i16.ll @@ -21,6 +21,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0(ptr addrspace(1) %out, ptr ad ; CIVI-LABEL: s_insertelement_v2i16_0: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CIVI-NEXT: v_mov_b32_e32 v0, s0 @@ -68,6 +71,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reg(ptr addrspace(1) %out, pt ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -84,6 +90,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reg(ptr addrspace(1) %out, pt ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0xc +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -152,6 +161,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_multi_use_hi_reg(ptr addrspac ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -172,6 +184,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_multi_use_hi_reg(ptr addrspac ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0xc +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -251,6 +266,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reghi(ptr addrspace(1) %out, ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -266,6 +284,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reghi(ptr addrspace(1) %out, ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0xc +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -320,6 +341,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reghi_multi_use_1(ptr addrspa ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -339,6 +363,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reghi_multi_use_1(ptr addrspa ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -407,6 +434,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reghi_both_multi_use_1(ptr ad ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -429,6 +459,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_0_reghi_both_multi_use_1(ptr ad ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -498,6 +531,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_1(ptr addrspace(1) %out, ptr ad ; CIVI-LABEL: s_insertelement_v2i16_1: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CIVI-NEXT: v_mov_b32_e32 v0, s0 @@ -544,6 +580,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_1_reg(ptr addrspace(1) %out, pt ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -560,6 +599,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_1_reg(ptr addrspace(1) %out, pt ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0xc +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -623,6 +665,9 @@ define amdgpu_kernel void @s_insertelement_v2f16_0(ptr addrspace(1) %out, ptr ad ; CIVI-LABEL: s_insertelement_v2f16_0: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CIVI-NEXT: v_mov_b32_e32 v0, s0 @@ -668,6 +713,9 @@ define amdgpu_kernel void @s_insertelement_v2f16_1(ptr addrspace(1) %out, ptr ad ; CIVI-LABEL: s_insertelement_v2f16_1: ; CIVI: ; %bb.0: ; CIVI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CIVI-NEXT: s_add_i32 s12, s12, s17 +; CIVI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CIVI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CIVI-NEXT: s_waitcnt lgkmcnt(0) ; CIVI-NEXT: s_load_dword s2, s[2:3], 0x0 ; CIVI-NEXT: v_mov_b32_e32 v0, s0 @@ -714,6 +762,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_0(ptr addrspace(1) %out, ptr ad ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -732,6 +783,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_0(ptr addrspace(1) %out, ptr ad ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -788,9 +842,12 @@ define amdgpu_kernel void @v_insertelement_v2i16_0_reghi(ptr addrspace(1) %out, ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dword v3, v[0:1] ; VI-NEXT: v_mov_b32_e32 v1, s1 @@ -807,9 +864,12 @@ define amdgpu_kernel void @v_insertelement_v2i16_0_reghi(ptr addrspace(1) %out, ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dword v3, v[0:1] ; CI-NEXT: v_mov_b32_e32 v1, s1 @@ -880,6 +940,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_0_inlineimm(ptr addrspace(1) %o ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -898,6 +961,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_0_inlineimm(ptr addrspace(1) %o ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -953,6 +1019,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_1(ptr addrspace(1) %out, ptr ad ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -971,6 +1040,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_1(ptr addrspace(1) %out, ptr ad ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -1038,6 +1110,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_1_inlineimm(ptr addrspace(1) %o ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -1056,6 +1131,9 @@ define amdgpu_kernel void @v_insertelement_v2i16_1_inlineimm(ptr addrspace(1) %o ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -1123,6 +1201,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_0(ptr addrspace(1) %out, ptr ad ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -1141,6 +1222,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_0(ptr addrspace(1) %out, ptr ad ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -1195,6 +1279,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_0_inlineimm(ptr addrspace(1) %o ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -1213,6 +1300,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_0_inlineimm(ptr addrspace(1) %o ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -1267,6 +1357,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_1(ptr addrspace(1) %out, ptr ad ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -1285,6 +1378,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_1(ptr addrspace(1) %out, ptr ad ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -1353,6 +1449,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_1_inlineimm(ptr addrspace(1) %o ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 @@ -1371,6 +1470,9 @@ define amdgpu_kernel void @v_insertelement_v2f16_1_inlineimm(ptr addrspace(1) %o ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 @@ -1445,6 +1547,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_dynamic(ptr addrspace(1) %out, ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s4, s[4:5], 0x0 ; VI-NEXT: s_load_dword s2, s[2:3], 0x0 @@ -1464,6 +1569,9 @@ define amdgpu_kernel void @s_insertelement_v2i16_dynamic(ptr addrspace(1) %out, ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s4, s[4:5], 0x0 ; CI-NEXT: s_load_dword s2, s[2:3], 0x0 @@ -1526,9 +1634,12 @@ define amdgpu_kernel void @v_insertelement_v2i16_dynamic_sgpr(ptr addrspace(1) % ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dword v3, v[0:1] ; VI-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -1547,9 +1658,12 @@ define amdgpu_kernel void @v_insertelement_v2i16_dynamic_sgpr(ptr addrspace(1) % ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dword v3, v[0:1] ; CI-NEXT: v_add_i32_e32 v0, vcc, s0, v2 @@ -1612,11 +1726,14 @@ define amdgpu_kernel void @v_insertelement_v2f16_dynamic_vgpr(ptr addrspace(1) % ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) -; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: v_mov_b32_e32 v1, s5 ; VI-NEXT: v_add_u32_e32 v0, vcc, s4, v2 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: v_mov_b32_e32 v3, s3 ; VI-NEXT: flat_load_dword v4, v[0:1] ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v3, vcc @@ -1639,11 +1756,14 @@ define amdgpu_kernel void @v_insertelement_v2f16_dynamic_vgpr(ptr addrspace(1) % ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) -; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: v_mov_b32_e32 v1, s5 ; CI-NEXT: v_add_i32_e32 v0, vcc, s4, v2 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: v_mov_b32_e32 v3, s3 ; CI-NEXT: flat_load_dword v4, v[0:1] ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v3, vcc @@ -1712,14 +1832,17 @@ define amdgpu_kernel void @v_insertelement_v4f16_0(ptr addrspace(1) %out, ptr ad ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v0, s4, v0, v4 @@ -1731,9 +1854,12 @@ define amdgpu_kernel void @v_insertelement_v4f16_0(ptr addrspace(1) %out, ptr ad ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0xc ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; CI-NEXT: v_mov_b32_e32 v3, s1 @@ -1790,14 +1916,17 @@ define amdgpu_kernel void @v_insertelement_v4f16_1(ptr addrspace(1) %out, ptr ad ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v0, v0, s4, v4 @@ -1809,9 +1938,12 @@ define amdgpu_kernel void @v_insertelement_v4f16_1(ptr addrspace(1) %out, ptr ad ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; CI-NEXT: v_mov_b32_e32 v3, s1 @@ -1883,14 +2015,17 @@ define amdgpu_kernel void @v_insertelement_v4f16_2(ptr addrspace(1) %out, ptr ad ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x30 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v1, s4, v1, v4 @@ -1902,9 +2037,12 @@ define amdgpu_kernel void @v_insertelement_v4f16_2(ptr addrspace(1) %out, ptr ad ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0xc ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; CI-NEXT: v_mov_b32_e32 v3, s1 @@ -1961,14 +2099,17 @@ define amdgpu_kernel void @v_insertelement_v4f16_3(ptr addrspace(1) %out, ptr ad ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x1000504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v1, v1, s4, v4 @@ -1980,9 +2121,12 @@ define amdgpu_kernel void @v_insertelement_v4f16_3(ptr addrspace(1) %out, ptr ad ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; CI-NEXT: v_mov_b32_e32 v3, s1 @@ -2054,14 +2198,17 @@ define amdgpu_kernel void @v_insertelement_v4i16_2(ptr addrspace(1) %out, ptr ad ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 -; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 ; VI-NEXT: v_add_u32_e32 v2, vcc, s0, v2 +; VI-NEXT: v_mov_b32_e32 v4, 0x3020504 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_perm_b32 v1, s4, v1, v4 @@ -2073,9 +2220,12 @@ define amdgpu_kernel void @v_insertelement_v4i16_2(ptr addrspace(1) %out, ptr ad ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; CI-NEXT: v_mov_b32_e32 v3, s1 @@ -2138,6 +2288,9 @@ define amdgpu_kernel void @v_insertelement_v4i16_dynamic_vgpr(ptr addrspace(1) % ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: flat_load_dword v4, v[0:1] glc ; VI-NEXT: s_waitcnt vmcnt(0) ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 @@ -2165,6 +2318,9 @@ define amdgpu_kernel void @v_insertelement_v4i16_dynamic_vgpr(ptr addrspace(1) % ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: flat_load_dword v4, v[0:1] glc ; CI-NEXT: s_waitcnt vmcnt(0) ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 @@ -2268,9 +2424,12 @@ define amdgpu_kernel void @v_insertelement_v4f16_dynamic_sgpr(ptr addrspace(1) % ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v2 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: v_mov_b32_e32 v3, s1 @@ -2294,9 +2453,12 @@ define amdgpu_kernel void @v_insertelement_v4f16_dynamic_sgpr(ptr addrspace(1) % ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; CI-NEXT: v_mov_b32_e32 v3, s1 @@ -2363,9 +2525,12 @@ define amdgpu_kernel void @v_insertelement_v8f16_3(ptr addrspace(1) %out, ptr ad ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: v_add_u32_e32 v4, vcc, s0, v4 @@ -2383,9 +2548,12 @@ define amdgpu_kernel void @v_insertelement_v8f16_3(ptr addrspace(1) %out, ptr ad ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; CI-NEXT: v_mov_b32_e32 v5, s1 @@ -2457,9 +2625,12 @@ define amdgpu_kernel void @v_insertelement_v8i16_6(ptr addrspace(1) %out, ptr ad ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: v_mov_b32_e32 v5, s1 @@ -2477,9 +2648,12 @@ define amdgpu_kernel void @v_insertelement_v8i16_6(ptr addrspace(1) %out, ptr ad ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; CI-NEXT: v_mov_b32_e32 v5, s1 @@ -2568,9 +2742,12 @@ define amdgpu_kernel void @v_insertelement_v8f16_dynamic(ptr addrspace(1) %out, ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: v_mov_b32_e32 v5, s1 @@ -2622,9 +2799,12 @@ define amdgpu_kernel void @v_insertelement_v8f16_dynamic(ptr addrspace(1) %out, ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; CI-NEXT: v_mov_b32_e32 v5, s1 @@ -2803,11 +2983,14 @@ define amdgpu_kernel void @v_insertelement_v16f16_3(ptr addrspace(1) %out, ptr a ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v8, 5, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_add_u32_e32 v4, vcc, 16, v0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v5, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: flat_load_dwordx4 v[4:7], v[4:5] @@ -2830,9 +3013,12 @@ define amdgpu_kernel void @v_insertelement_v16f16_3(ptr addrspace(1) %out, ptr a ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v8, 5, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s3 ; CI-NEXT: v_add_i32_e32 v4, vcc, s2, v8 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v5, vcc, 0, v0, vcc ; CI-NEXT: flat_load_dwordx4 v[0:3], v[4:5] ; CI-NEXT: v_add_i32_e32 v4, vcc, 16, v4 @@ -2923,12 +3109,14 @@ define amdgpu_kernel void @v_insertelement_v16i16_6(ptr addrspace(1) %out, ptr a ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dword s4, s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v8, 5, v0 -; VI-NEXT: v_mov_b32_e32 v12, 0x3020504 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_add_u32_e32 v4, vcc, 16, v0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v5, vcc, 0, v1, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: flat_load_dwordx4 v[4:7], v[4:5] @@ -2936,6 +3124,7 @@ define amdgpu_kernel void @v_insertelement_v16i16_6(ptr addrspace(1) %out, ptr a ; VI-NEXT: v_add_u32_e32 v8, vcc, s0, v8 ; VI-NEXT: v_addc_u32_e32 v9, vcc, 0, v9, vcc ; VI-NEXT: v_add_u32_e32 v10, vcc, 16, v8 +; VI-NEXT: v_mov_b32_e32 v12, 0x3020504 ; VI-NEXT: v_addc_u32_e32 v11, vcc, 0, v9, vcc ; VI-NEXT: s_waitcnt vmcnt(1) ; VI-NEXT: v_perm_b32 v3, s4, v3, v12 @@ -2949,11 +3138,14 @@ define amdgpu_kernel void @v_insertelement_v16i16_6(ptr addrspace(1) %out, ptr a ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dword s4, s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v8, 5, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_add_i32_e32 v4, vcc, 16, v0 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v5, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; CI-NEXT: flat_load_dwordx4 v[4:7], v[4:5] @@ -3087,11 +3279,14 @@ define amdgpu_kernel void @v_insertelement_v16f16_dynamic(ptr addrspace(1) %out, ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v8, 5, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s3 ; VI-NEXT: v_add_u32_e32 v4, vcc, s2, v8 ; VI-NEXT: v_addc_u32_e32 v5, vcc, 0, v0, vcc ; VI-NEXT: v_add_u32_e32 v0, vcc, 16, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v5, vcc ; VI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] ; VI-NEXT: flat_load_dwordx4 v[4:7], v[4:5] @@ -3184,11 +3379,14 @@ define amdgpu_kernel void @v_insertelement_v16f16_dynamic(ptr addrspace(1) %out, ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 5, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_add_i32_e32 v2, vcc, 16, v0 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v3, vcc, 0, v1, vcc ; CI-NEXT: flat_load_dwordx4 v[7:10], v[2:3] ; CI-NEXT: flat_load_dwordx4 v[0:3], v[0:1] diff --git a/llvm/test/CodeGen/AMDGPU/invalid-addrspacecast.ll b/llvm/test/CodeGen/AMDGPU/invalid-addrspacecast.ll index f0609f62a9024..5dff7372ab561 100644 --- a/llvm/test/CodeGen/AMDGPU/invalid-addrspacecast.ll +++ b/llvm/test/CodeGen/AMDGPU/invalid-addrspacecast.ll @@ -7,6 +7,9 @@ define amdgpu_kernel void @use_group_to_global_addrspacecast(ptr addrspace(3) %ptr) { ; CHECK-LABEL: use_group_to_global_addrspacecast: ; CHECK: ; %bb.0: +; CHECK-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-NEXT: s_add_i32 s12, s12, s17 +; CHECK-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-NEXT: v_mov_b32_e32 v0, 0 ; CHECK-NEXT: flat_store_dword v[0:1], v0 ; CHECK-NEXT: s_waitcnt vmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/invalid-cast-load-i1.ll b/llvm/test/CodeGen/AMDGPU/invalid-cast-load-i1.ll index 621187100f323..55a5d50f06bbd 100644 --- a/llvm/test/CodeGen/AMDGPU/invalid-cast-load-i1.ll +++ b/llvm/test/CodeGen/AMDGPU/invalid-cast-load-i1.ll @@ -6,6 +6,8 @@ define amdgpu_kernel void @load_idx_idy(ptr addrspace(4) %disp, ptr %g) { ; CHECK: ; %bb.0: ; %entry ; CHECK-NEXT: s_load_dword s6, s[4:5], 0x4 ; CHECK-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; CHECK-NEXT: v_mov_b32_e32 v0, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: s_lshr_b32 s4, s6, 16 diff --git a/llvm/test/CodeGen/AMDGPU/kernarg-size.ll b/llvm/test/CodeGen/AMDGPU/kernarg-size.ll index 496a1c652da25..1a32953305bbc 100644 --- a/llvm/test/CodeGen/AMDGPU/kernarg-size.ll +++ b/llvm/test/CodeGen/AMDGPU/kernarg-size.ll @@ -7,7 +7,7 @@ declare void @llvm.trap() #0 ; DOORBELL-NEXT: .amdhsa_group_segment_fixed_size 0 ; DOORBELL-NEXT: .amdhsa_private_segment_fixed_size 0 ; DOORBELL-NEXT: .amdhsa_kernarg_size 8 -; DOORBELL-NEXT: .amdhsa_user_sgpr_count 12 +; DOORBELL-NEXT: .amdhsa_user_sgpr_count 14 ; DOORBELL-NEXT: .amdhsa_user_sgpr_private_segment_buffer 1 ; DOORBELL: .end_amdhsa_kernel diff --git a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll index 4b6cc32522f5b..7179f687c70f2 100644 --- a/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll +++ b/llvm/test/CodeGen/AMDGPU/llc-pipeline.ll @@ -47,11 +47,7 @@ ; GCN-O0-NEXT: Instrument function entry/exit with calls to e.g. mcount() (post inlining) ; GCN-O0-NEXT: Scalarize Masked Memory Intrinsics ; GCN-O0-NEXT: Expand reduction intrinsics -; GCN-O0-NEXT: CallGraph Construction -; GCN-O0-NEXT: Call Graph SCC Pass Manager -; GCN-O0-NEXT: AMDGPU Annotate Kernel Features -; GCN-O0-NEXT: FunctionPass Manager -; GCN-O0-NEXT: AMDGPU Lower Kernel Arguments +; GCN-O0-NEXT: AMDGPU Lower Kernel Arguments ; GCN-O0-NEXT: Lower buffer fat pointer operations to buffer resources ; GCN-O0-NEXT: CallGraph Construction ; GCN-O0-NEXT: Call Graph SCC Pass Manager @@ -232,11 +228,7 @@ ; GCN-O1-NEXT: Instrument function entry/exit with calls to e.g. mcount() (post inlining) ; GCN-O1-NEXT: Scalarize Masked Memory Intrinsics ; GCN-O1-NEXT: Expand reduction intrinsics -; GCN-O1-NEXT: CallGraph Construction -; GCN-O1-NEXT: Call Graph SCC Pass Manager -; GCN-O1-NEXT: AMDGPU Annotate Kernel Features -; GCN-O1-NEXT: FunctionPass Manager -; GCN-O1-NEXT: AMDGPU Lower Kernel Arguments +; GCN-O1-NEXT: AMDGPU Lower Kernel Arguments ; GCN-O1-NEXT: Lower buffer fat pointer operations to buffer resources ; GCN-O1-NEXT: CallGraph Construction ; GCN-O1-NEXT: Call Graph SCC Pass Manager @@ -531,11 +523,7 @@ ; GCN-O1-OPTS-NEXT: Scalarize Masked Memory Intrinsics ; GCN-O1-OPTS-NEXT: Expand reduction intrinsics ; GCN-O1-OPTS-NEXT: Early CSE -; GCN-O1-OPTS-NEXT: CallGraph Construction -; GCN-O1-OPTS-NEXT: Call Graph SCC Pass Manager -; GCN-O1-OPTS-NEXT: AMDGPU Annotate Kernel Features -; GCN-O1-OPTS-NEXT: FunctionPass Manager -; GCN-O1-OPTS-NEXT: AMDGPU Lower Kernel Arguments +; GCN-O1-OPTS-NEXT: AMDGPU Lower Kernel Arguments ; GCN-O1-OPTS-NEXT: Lower buffer fat pointer operations to buffer resources ; GCN-O1-OPTS-NEXT: CallGraph Construction ; GCN-O1-OPTS-NEXT: Call Graph SCC Pass Manager @@ -848,11 +836,7 @@ ; GCN-O2-NEXT: Scalarize Masked Memory Intrinsics ; GCN-O2-NEXT: Expand reduction intrinsics ; GCN-O2-NEXT: Early CSE -; GCN-O2-NEXT: CallGraph Construction -; GCN-O2-NEXT: Call Graph SCC Pass Manager -; GCN-O2-NEXT: AMDGPU Annotate Kernel Features -; GCN-O2-NEXT: FunctionPass Manager -; GCN-O2-NEXT: AMDGPU Lower Kernel Arguments +; GCN-O2-NEXT: AMDGPU Lower Kernel Arguments ; GCN-O2-NEXT: Lower buffer fat pointer operations to buffer resources ; GCN-O2-NEXT: CallGraph Construction ; GCN-O2-NEXT: Call Graph SCC Pass Manager @@ -1180,11 +1164,7 @@ ; GCN-O3-NEXT: Lazy Block Frequency Analysis ; GCN-O3-NEXT: Optimization Remark Emitter ; GCN-O3-NEXT: Global Value Numbering -; GCN-O3-NEXT: CallGraph Construction -; GCN-O3-NEXT: Call Graph SCC Pass Manager -; GCN-O3-NEXT: AMDGPU Annotate Kernel Features -; GCN-O3-NEXT: FunctionPass Manager -; GCN-O3-NEXT: AMDGPU Lower Kernel Arguments +; GCN-O3-NEXT: AMDGPU Lower Kernel Arguments ; GCN-O3-NEXT: Lower buffer fat pointer operations to buffer resources ; GCN-O3-NEXT: CallGraph Construction ; GCN-O3-NEXT: Call Graph SCC Pass Manager diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.private.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.private.ll index f93d80cc7adf8..4edd0357c6e7a 100644 --- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.private.ll +++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.private.ll @@ -30,9 +30,12 @@ define amdgpu_kernel void @is_private_vgpr(ptr addrspace(1) %ptr.ptr) { ; CI-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-SDAG-NEXT: s_load_dword s2, s[8:9], 0x32 ; CI-SDAG-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; CI-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CI-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CI-SDAG-NEXT: v_mov_b32_e32 v1, s1 ; CI-SDAG-NEXT: v_add_i32_e32 v0, vcc, s0, v0 +; CI-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-SDAG-NEXT: flat_load_dwordx2 v[0:1], v[0:1] glc ; CI-SDAG-NEXT: s_waitcnt vmcnt(0) @@ -59,10 +62,13 @@ define amdgpu_kernel void @is_private_vgpr(ptr addrspace(1) %ptr.ptr) { ; CI-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-GISEL-NEXT: s_load_dword s2, s[8:9], 0x32 ; CI-GISEL-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CI-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CI-GISEL-NEXT: v_mov_b32_e32 v0, s0 ; CI-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; CI-GISEL-NEXT: v_add_i32_e32 v0, vcc, v0, v2 +; CI-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-GISEL-NEXT: flat_load_dwordx2 v[0:1], v[0:1] glc ; CI-GISEL-NEXT: s_waitcnt vmcnt(0) @@ -133,6 +139,9 @@ define amdgpu_kernel void @is_private_sgpr(ptr %ptr) { ; CI-SDAG: ; %bb.0: ; CI-SDAG-NEXT: s_load_dword s0, s[8:9], 0x1 ; CI-SDAG-NEXT: s_load_dword s1, s[8:9], 0x32 +; CI-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CI-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CI-SDAG-NEXT: s_cmp_eq_u32 s0, s1 ; CI-SDAG-NEXT: s_cselect_b64 s[0:1], -1, 0 @@ -166,6 +175,9 @@ define amdgpu_kernel void @is_private_sgpr(ptr %ptr) { ; CI-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CI-GISEL-NEXT: s_load_dword s0, s[8:9], 0x32 +; CI-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CI-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CI-GISEL-NEXT: s_cmp_lg_u32 s1, s0 ; CI-GISEL-NEXT: s_cbranch_scc1 .LBB1_2 diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.shared.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.shared.ll index 637d8388cddf1..9d078f7906b4d 100644 --- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.shared.ll +++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.is.shared.ll @@ -63,9 +63,12 @@ define amdgpu_kernel void @is_local_vgpr(ptr addrspace(1) %ptr.ptr) { ; CI-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-SDAG-NEXT: s_load_dword s2, s[8:9], 0x33 ; CI-SDAG-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; CI-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CI-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CI-SDAG-NEXT: v_mov_b32_e32 v1, s1 ; CI-SDAG-NEXT: v_add_i32_e32 v0, vcc, s0, v0 +; CI-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-SDAG-NEXT: flat_load_dwordx2 v[0:1], v[0:1] glc ; CI-SDAG-NEXT: s_waitcnt vmcnt(0) @@ -92,10 +95,13 @@ define amdgpu_kernel void @is_local_vgpr(ptr addrspace(1) %ptr.ptr) { ; CI-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-GISEL-NEXT: s_load_dword s2, s[8:9], 0x33 ; CI-GISEL-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CI-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CI-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CI-GISEL-NEXT: v_mov_b32_e32 v0, s0 ; CI-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; CI-GISEL-NEXT: v_add_i32_e32 v0, vcc, v0, v2 +; CI-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-GISEL-NEXT: flat_load_dwordx2 v[0:1], v[0:1] glc ; CI-GISEL-NEXT: s_waitcnt vmcnt(0) @@ -200,6 +206,9 @@ define amdgpu_kernel void @is_local_sgpr(ptr %ptr) { ; CI-SDAG: ; %bb.0: ; CI-SDAG-NEXT: s_load_dword s0, s[8:9], 0x1 ; CI-SDAG-NEXT: s_load_dword s1, s[8:9], 0x33 +; CI-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CI-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CI-SDAG-NEXT: s_cmp_eq_u32 s0, s1 ; CI-SDAG-NEXT: s_cselect_b64 s[0:1], -1, 0 @@ -233,6 +242,9 @@ define amdgpu_kernel void @is_local_sgpr(ptr %ptr) { ; CI-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CI-GISEL-NEXT: s_load_dword s0, s[8:9], 0x33 +; CI-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CI-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CI-GISEL-NEXT: s_cmp_lg_u32 s1, s0 ; CI-GISEL-NEXT: s_cbranch_scc1 .LBB1_2 diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.lds.kernel.id.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.lds.kernel.id.ll index 97219a8f143ce..0fe371c1b51fe 100644 --- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.lds.kernel.id.ll +++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.lds.kernel.id.ll @@ -23,8 +23,11 @@ define void @function_lds_id(ptr addrspace(1) %out) { define amdgpu_kernel void @kernel_lds_id(ptr addrspace(1) %out) !llvm.amdgcn.lds.kernel.id !0 { ; GCN-LABEL: kernel_lds_id: ; GCN: ; %bb.0: +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; GCN-NEXT: s_add_i32 s2, s12, 42 +; GCN-NEXT: s_add_i32 s2, s14, 42 ; GCN-NEXT: v_mov_b32_e32 v2, s2 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s0 @@ -74,6 +77,9 @@ define amdgpu_kernel void @indirect_lds_id(ptr addrspace(1) %out) !llvm.amdgcn.l define amdgpu_kernel void @doesnt_use_it(ptr addrspace(1) %out) !llvm.amdgcn.lds.kernel.id !0 { ; GCN-LABEL: doesnt_use_it: ; GCN: ; %bb.0: +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GCN-NEXT: v_mov_b32_e32 v2, 0x64 ; GCN-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readfirstlane.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readfirstlane.ll index 55fa02a0c582c..cc9e34be209b4 100644 --- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readfirstlane.ll +++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readfirstlane.ll @@ -284,6 +284,9 @@ define amdgpu_kernel void @test_readfirstlane_imm_fold_i32(ptr addrspace(1) %out ; CHECK-SDAG-LABEL: test_readfirstlane_imm_fold_i32: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, 32 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s0 @@ -294,6 +297,9 @@ define amdgpu_kernel void @test_readfirstlane_imm_fold_i32(ptr addrspace(1) %out ; CHECK-GISEL-LABEL: test_readfirstlane_imm_fold_i32: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, 32 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s0 @@ -309,10 +315,13 @@ define amdgpu_kernel void @test_readfirstlane_imm_fold_i64(ptr addrspace(1) %out ; CHECK-SDAG-LABEL: test_readfirstlane_imm_fold_i64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, 32 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -321,10 +330,13 @@ define amdgpu_kernel void @test_readfirstlane_imm_fold_i64(ptr addrspace(1) %out ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CHECK-GISEL-NEXT: s_mov_b64 s[2:3], 32 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm @@ -337,10 +349,13 @@ define amdgpu_kernel void @test_readfirstlane_imm_fold_f64(ptr addrspace(1) %out ; CHECK-SDAG-LABEL: test_readfirstlane_imm_fold_f64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, 0 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0x40400000 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0x40400000 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -349,11 +364,14 @@ define amdgpu_kernel void @test_readfirstlane_imm_fold_f64(ptr addrspace(1) %out ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CHECK-GISEL-NEXT: s_mov_b32 s2, 0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: s_mov_b32 s3, 0x40400000 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm @@ -366,12 +384,15 @@ define amdgpu_kernel void @test_readfirstlane_m0(ptr addrspace(1) %out) { ; CHECK-SDAG-LABEL: test_readfirstlane_m0: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b32 m0, -1 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s1 ; CHECK-SDAG-NEXT: flat_store_dword v[0:1], v2 ; CHECK-SDAG-NEXT: s_endpgm @@ -379,12 +400,15 @@ define amdgpu_kernel void @test_readfirstlane_m0(ptr addrspace(1) %out) { ; CHECK-GISEL-LABEL: test_readfirstlane_m0: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b32 m0, -1 ; CHECK-GISEL-NEXT: ;;#ASMEND -; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; CHECK-GISEL-NEXT: flat_store_dword v[0:1], v2 ; CHECK-GISEL-NEXT: s_endpgm @@ -398,25 +422,31 @@ define amdgpu_kernel void @test_readfirstlane_copy_from_sgpr_i32(ptr addrspace(1 ; CHECK-SDAG-LABEL: test_readfirstlane_copy_from_sgpr_i32: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b32 s2, 0 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s0 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-SDAG-NEXT: flat_store_dword v[0:1], v2 ; CHECK-SDAG-NEXT: s_endpgm ; ; CHECK-GISEL-LABEL: test_readfirstlane_copy_from_sgpr_i32: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b32 s2, 0 ; CHECK-GISEL-NEXT: ;;#ASMEND -; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; CHECK-GISEL-NEXT: flat_store_dword v[0:1], v2 ; CHECK-GISEL-NEXT: s_endpgm @@ -430,13 +460,16 @@ define amdgpu_kernel void @test_readfirstlane_copy_from_sgpr_i64(ptr addrspace(1 ; CHECK-SDAG-LABEL: test_readfirstlane_copy_from_sgpr_i64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -444,13 +477,16 @@ define amdgpu_kernel void @test_readfirstlane_copy_from_sgpr_i64(ptr addrspace(1 ; CHECK-GISEL-LABEL: test_readfirstlane_copy_from_sgpr_i64: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-GISEL-NEXT: ;;#ASMEND ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm @@ -464,13 +500,16 @@ define amdgpu_kernel void @test_readfirstlane_copy_from_sgpr_f64(ptr addrspace(1 ; CHECK-SDAG-LABEL: test_readfirstlane_copy_from_sgpr_f64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -478,13 +517,16 @@ define amdgpu_kernel void @test_readfirstlane_copy_from_sgpr_f64(ptr addrspace(1 ; CHECK-GISEL-LABEL: test_readfirstlane_copy_from_sgpr_f64: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-GISEL-NEXT: ;;#ASMEND ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readlane.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readlane.ll index edb6ebcee1325..f2b0959cc706e 100644 --- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readlane.ll +++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.readlane.ll @@ -179,6 +179,9 @@ define amdgpu_kernel void @test_readlane_imm_sreg_i32(ptr addrspace(1) %out, i32 ; CHECK-SDAG-LABEL: test_readlane_imm_sreg_i32: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, 32 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s0 @@ -189,6 +192,9 @@ define amdgpu_kernel void @test_readlane_imm_sreg_i32(ptr addrspace(1) %out, i32 ; CHECK-GISEL-LABEL: test_readlane_imm_sreg_i32: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, 32 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s0 @@ -204,10 +210,13 @@ define amdgpu_kernel void @test_readlane_imm_sreg_i64(ptr addrspace(1) %out, i32 ; CHECK-SDAG-LABEL: test_readlane_imm_sreg_i64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, 32 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -216,10 +225,13 @@ define amdgpu_kernel void @test_readlane_imm_sreg_i64(ptr addrspace(1) %out, i32 ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CHECK-GISEL-NEXT: s_mov_b64 s[2:3], 32 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm @@ -232,10 +244,13 @@ define amdgpu_kernel void @test_readlane_imm_sreg_f64(ptr addrspace(1) %out, i32 ; CHECK-SDAG-LABEL: test_readlane_imm_sreg_f64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, 0 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0x40400000 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, 0x40400000 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -244,11 +259,14 @@ define amdgpu_kernel void @test_readlane_imm_sreg_f64(ptr addrspace(1) %out, i32 ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CHECK-GISEL-NEXT: s_mov_b32 s2, 0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: s_mov_b32 s3, 0x40400000 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm @@ -262,6 +280,9 @@ define amdgpu_kernel void @test_readlane_vregs_i32(ptr addrspace(1) %out, ptr ad ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CHECK-SDAG-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -281,6 +302,9 @@ define amdgpu_kernel void @test_readlane_vregs_i32(ptr addrspace(1) %out, ptr ad ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CHECK-GISEL-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 @@ -311,6 +335,9 @@ define amdgpu_kernel void @test_readlane_vregs_i64(ptr addrspace(1) %out, ptr ad ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CHECK-SDAG-NEXT: v_lshlrev_b32_e32 v0, 4, v0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -332,6 +359,9 @@ define amdgpu_kernel void @test_readlane_vregs_i64(ptr addrspace(1) %out, ptr ad ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CHECK-GISEL-NEXT: v_lshlrev_b32_e32 v2, 4, v0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 @@ -365,6 +395,9 @@ define amdgpu_kernel void @test_readlane_vregs_f64(ptr addrspace(1) %out, ptr ad ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CHECK-SDAG-NEXT: v_lshlrev_b32_e32 v0, 4, v0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -386,6 +419,9 @@ define amdgpu_kernel void @test_readlane_vregs_f64(ptr addrspace(1) %out, ptr ad ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CHECK-GISEL-NEXT: v_lshlrev_b32_e32 v2, 4, v0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 @@ -419,12 +455,15 @@ define amdgpu_kernel void @test_readlane_m0_sreg(ptr addrspace(1) %out, i32 %src ; CHECK-SDAG-LABEL: test_readlane_m0_sreg: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b32 m0, -1 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s1 ; CHECK-SDAG-NEXT: flat_store_dword v[0:1], v2 ; CHECK-SDAG-NEXT: s_endpgm @@ -432,12 +471,15 @@ define amdgpu_kernel void @test_readlane_m0_sreg(ptr addrspace(1) %out, i32 %src ; CHECK-GISEL-LABEL: test_readlane_m0_sreg: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b32 m0, -1 ; CHECK-GISEL-NEXT: ;;#ASMEND -; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, m0 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; CHECK-GISEL-NEXT: flat_store_dword v[0:1], v2 ; CHECK-GISEL-NEXT: s_endpgm @@ -454,11 +496,14 @@ define amdgpu_kernel void @test_readlane_vgpr_imm_i32(ptr addrspace(1) %out) #1 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: ; def v0 ; CHECK-SDAG-NEXT: ;;#ASMEND +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 ; CHECK-SDAG-NEXT: v_readlane_b32 s2, v0, 32 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s2 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-SDAG-NEXT: flat_store_dword v[0:1], v2 ; CHECK-SDAG-NEXT: s_endpgm ; @@ -468,10 +513,13 @@ define amdgpu_kernel void @test_readlane_vgpr_imm_i32(ptr addrspace(1) %out) #1 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: ; def v0 ; CHECK-GISEL-NEXT: ;;#ASMEND +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: v_readlane_b32 s2, v0, 32 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s2 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; CHECK-GISEL-NEXT: flat_store_dword v[0:1], v2 ; CHECK-GISEL-NEXT: s_endpgm @@ -485,14 +533,17 @@ define amdgpu_kernel void @test_readlane_vgpr_imm_i64(ptr addrspace(1) %out) #1 ; CHECK-SDAG-LABEL: test_readlane_vgpr_imm_i64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: ; def v[0:1] ; CHECK-SDAG-NEXT: ;;#ASMEND ; CHECK-SDAG-NEXT: v_readlane_b32 s2, v1, 32 ; CHECK-SDAG-NEXT: v_readlane_b32 s3, v0, 32 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s3 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s3 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s2 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -505,10 +556,13 @@ define amdgpu_kernel void @test_readlane_vgpr_imm_i64(ptr addrspace(1) %out) #1 ; CHECK-GISEL-NEXT: ; def v[0:1] ; CHECK-GISEL-NEXT: ;;#ASMEND ; CHECK-GISEL-NEXT: v_readlane_b32 s2, v0, 32 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: v_readlane_b32 s3, v1, 32 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -523,14 +577,17 @@ define amdgpu_kernel void @test_readlane_vgpr_imm_f64(ptr addrspace(1) %out) #1 ; CHECK-SDAG-LABEL: test_readlane_vgpr_imm_f64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: ; def v[0:1] ; CHECK-SDAG-NEXT: ;;#ASMEND ; CHECK-SDAG-NEXT: v_readlane_b32 s2, v1, 32 ; CHECK-SDAG-NEXT: v_readlane_b32 s3, v0, 32 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s3 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s3 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s2 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -543,10 +600,13 @@ define amdgpu_kernel void @test_readlane_vgpr_imm_f64(ptr addrspace(1) %out) #1 ; CHECK-GISEL-NEXT: ; def v[0:1] ; CHECK-GISEL-NEXT: ;;#ASMEND ; CHECK-GISEL-NEXT: v_readlane_b32 s2, v0, 32 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: v_readlane_b32 s3, v1, 32 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -561,25 +621,31 @@ define amdgpu_kernel void @test_readlane_copy_from_sgpr_i32(ptr addrspace(1) %ou ; CHECK-SDAG-LABEL: test_readlane_copy_from_sgpr_i32: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b32 s2, 0 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s0 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-SDAG-NEXT: flat_store_dword v[0:1], v2 ; CHECK-SDAG-NEXT: s_endpgm ; ; CHECK-GISEL-LABEL: test_readlane_copy_from_sgpr_i32: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b32 s2, 0 ; CHECK-GISEL-NEXT: ;;#ASMEND -; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s0 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s2 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; CHECK-GISEL-NEXT: flat_store_dword v[0:1], v2 ; CHECK-GISEL-NEXT: s_endpgm @@ -593,13 +659,16 @@ define amdgpu_kernel void @test_readlane_copy_from_sgpr_i64(ptr addrspace(1) %ou ; CHECK-SDAG-LABEL: test_readlane_copy_from_sgpr_i64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -607,13 +676,16 @@ define amdgpu_kernel void @test_readlane_copy_from_sgpr_i64(ptr addrspace(1) %ou ; CHECK-GISEL-LABEL: test_readlane_copy_from_sgpr_i64: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-GISEL-NEXT: ;;#ASMEND ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm @@ -627,13 +699,16 @@ define amdgpu_kernel void @test_readlane_copy_from_sgpr_f64(ptr addrspace(1) %ou ; CHECK-SDAG-LABEL: test_readlane_copy_from_sgpr_f64: ; CHECK-SDAG: ; %bb.0: ; CHECK-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-SDAG-NEXT: s_add_i32 s12, s12, s17 +; CHECK-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CHECK-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CHECK-SDAG-NEXT: ;;#ASMSTART ; CHECK-SDAG-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-SDAG-NEXT: ;;#ASMEND -; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v0, s2 +; CHECK-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-SDAG-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-SDAG-NEXT: s_endpgm @@ -641,13 +716,16 @@ define amdgpu_kernel void @test_readlane_copy_from_sgpr_f64(ptr addrspace(1) %ou ; CHECK-GISEL-LABEL: test_readlane_copy_from_sgpr_f64: ; CHECK-GISEL: ; %bb.0: ; CHECK-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CHECK-GISEL-NEXT: s_add_i32 s12, s12, s17 ; CHECK-GISEL-NEXT: ;;#ASMSTART ; CHECK-GISEL-NEXT: s_mov_b64 s[2:3], 0 ; CHECK-GISEL-NEXT: ;;#ASMEND ; CHECK-GISEL-NEXT: v_mov_b32_e32 v0, s2 -; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 +; CHECK-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CHECK-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; CHECK-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CHECK-GISEL-NEXT: v_mov_b32_e32 v1, s3 ; CHECK-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; CHECK-GISEL-NEXT: flat_store_dwordx2 v[2:3], v[0:1] ; CHECK-GISEL-NEXT: s_endpgm diff --git a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.writelane.ll b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.writelane.ll index 04d179478590b..4ac2cc98970b5 100644 --- a/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.writelane.ll +++ b/llvm/test/CodeGen/AMDGPU/llvm.amdgcn.writelane.ll @@ -15,6 +15,9 @@ define amdgpu_kernel void @test_writelane_sreg_i32(ptr addrspace(1) %out, i32 %s ; GFX802-SDAG-LABEL: test_writelane_sreg_i32: ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_mov_b32 m0, s3 ; GFX802-SDAG-NEXT: s_load_dword s3, s[0:1], 0x0 @@ -53,6 +56,9 @@ define amdgpu_kernel void @test_writelane_sreg_i32(ptr addrspace(1) %out, i32 %s ; GFX802-GISEL-LABEL: test_writelane_sreg_i32: ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_mov_b32 m0, s3 ; GFX802-GISEL-NEXT: s_load_dword s3, s[0:1], 0x0 @@ -98,6 +104,9 @@ define amdgpu_kernel void @test_writelane_sreg_i64(ptr addrspace(1) %out, i64 %s ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s6, s[8:9], 0x10 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s6 @@ -147,6 +156,9 @@ define amdgpu_kernel void @test_writelane_sreg_i64(ptr addrspace(1) %out, i64 %s ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s6, s[8:9], 0x10 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-GISEL-NEXT: s_mov_b32 m0, s6 @@ -202,6 +214,9 @@ define amdgpu_kernel void @test_writelane_sreg_f64(ptr addrspace(1) %out, double ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s6, s[8:9], 0x10 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s6 @@ -251,6 +266,9 @@ define amdgpu_kernel void @test_writelane_sreg_f64(ptr addrspace(1) %out, double ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s6, s[8:9], 0x10 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-GISEL-NEXT: s_mov_b32 m0, s6 @@ -306,6 +324,9 @@ define amdgpu_kernel void @test_writelane_imm_sreg_i32(ptr addrspace(1) %out, i3 ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s2, s[8:9], 0x8 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dword s3, s[0:1], 0x0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, s0 @@ -348,6 +369,9 @@ define amdgpu_kernel void @test_writelane_imm_sreg_i32(ptr addrspace(1) %out, i3 ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s2, s[8:9], 0x8 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dword s3, s[0:1], 0x0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s0 @@ -396,6 +420,9 @@ define amdgpu_kernel void @test_writelane_imm_sreg_i64(ptr addrspace(1) %out, i3 ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s4, s[8:9], 0x8 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dwordx2 s[2:3], s[0:1], 0x0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 @@ -444,6 +471,9 @@ define amdgpu_kernel void @test_writelane_imm_sreg_i64(ptr addrspace(1) %out, i3 ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s4, s[8:9], 0x8 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dwordx2 s[2:3], s[0:1], 0x0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s1 @@ -498,11 +528,14 @@ define amdgpu_kernel void @test_writelane_imm_sreg_f64(ptr addrspace(1) %out, i3 ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s4, s[8:9], 0x8 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 ; GFX802-SDAG-NEXT: s_mov_b32 s5, 0x40400000 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dwordx2 s[2:3], s[0:1], 0x0 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s4 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s3 @@ -551,11 +584,14 @@ define amdgpu_kernel void @test_writelane_imm_sreg_f64(ptr addrspace(1) %out, i3 ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s4, s[8:9], 0x8 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 ; GFX802-GISEL-NEXT: s_mov_b32 s5, 0x40400000 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dwordx2 s[2:3], s[0:1], 0x0 ; GFX802-GISEL-NEXT: s_mov_b32 m0, s4 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, s0 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s2 @@ -609,6 +645,9 @@ define amdgpu_kernel void @test_writelane_vreg_lane_i32(ptr addrspace(1) %out, p ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; GFX802-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -668,6 +707,9 @@ define amdgpu_kernel void @test_writelane_vreg_lane_i32(ptr addrspace(1) %out, p ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-GISEL-NEXT: v_lshlrev_b32_e32 v2, 3, v0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, s3 @@ -738,6 +780,9 @@ define amdgpu_kernel void @test_writelane_vreg_lane_i64(ptr addrspace(1) %out, p ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: v_lshlrev_b32_e32 v0, 4, v0 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; GFX802-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -803,6 +848,9 @@ define amdgpu_kernel void @test_writelane_vreg_lane_i64(ptr addrspace(1) %out, p ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-GISEL-NEXT: v_lshlrev_b32_e32 v2, 4, v0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, s3 @@ -877,7 +925,9 @@ define amdgpu_kernel void @test_writelane_vreg_lane_f64(ptr addrspace(1) %out, p ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: v_lshlrev_b32_e32 v0, 4, v0 -; GFX802-SDAG-NEXT: s_mov_b32 s4, 0x40280000 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; GFX802-SDAG-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -886,6 +936,7 @@ define amdgpu_kernel void @test_writelane_vreg_lane_f64(ptr addrspace(1) %out, p ; GFX802-SDAG-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; GFX802-SDAG-NEXT: flat_load_dword v2, v[0:1] ; GFX802-SDAG-NEXT: s_load_dwordx2 s[2:3], s[0:1], 0x0 +; GFX802-SDAG-NEXT: s_mov_b32 s4, 0x40280000 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, s2 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s3 @@ -946,7 +997,9 @@ define amdgpu_kernel void @test_writelane_vreg_lane_f64(ptr addrspace(1) %out, p ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-GISEL-NEXT: v_lshlrev_b32_e32 v2, 4, v0 -; GFX802-GISEL-NEXT: s_mov_b32 s4, 0x40280000 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s2 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, s3 @@ -956,6 +1009,7 @@ define amdgpu_kernel void @test_writelane_vreg_lane_f64(ptr addrspace(1) %out, p ; GFX802-GISEL-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; GFX802-GISEL-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; GFX802-GISEL-NEXT: s_load_dwordx2 s[2:3], s[0:1], 0x0 +; GFX802-GISEL-NEXT: s_mov_b32 s4, 0x40280000 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v4, s1 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s0 ; GFX802-GISEL-NEXT: s_waitcnt vmcnt(0) lgkmcnt(0) @@ -1028,15 +1082,18 @@ define amdgpu_kernel void @test_writelane_m0_sreg_i32(ptr addrspace(1) %out, i32 ; GFX802-SDAG-NEXT: ;;#ASMSTART ; GFX802-SDAG-NEXT: s_mov_b32 m0, -1 ; GFX802-SDAG-NEXT: ;;#ASMEND +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 ; GFX802-SDAG-NEXT: s_mov_b32 s4, m0 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dword s3, s[0:1], 0x0 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s2 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, s0 -; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s1 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s3 ; GFX802-SDAG-NEXT: v_writelane_b32 v2, s4, m0 +; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s1 ; GFX802-SDAG-NEXT: flat_store_dword v[0:1], v2 ; GFX802-SDAG-NEXT: s_endpgm ; @@ -1081,15 +1138,18 @@ define amdgpu_kernel void @test_writelane_m0_sreg_i32(ptr addrspace(1) %out, i32 ; GFX802-GISEL-NEXT: ;;#ASMSTART ; GFX802-GISEL-NEXT: s_mov_b32 m0, -1 ; GFX802-GISEL-NEXT: ;;#ASMEND +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 ; GFX802-GISEL-NEXT: s_mov_b32 s4, m0 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dword s3, s[0:1], 0x0 ; GFX802-GISEL-NEXT: s_mov_b32 m0, s2 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s0 -; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, s1 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, s3 ; GFX802-GISEL-NEXT: v_writelane_b32 v2, s4, m0 +; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; GFX802-GISEL-NEXT: flat_store_dword v[0:1], v2 ; GFX802-GISEL-NEXT: s_endpgm ; @@ -1138,6 +1198,9 @@ define amdgpu_kernel void @test_writelane_imm_i32(ptr addrspace(1) %out, i32 %sr ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s2, s[8:9], 0x8 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dword s3, s[0:1], 0x0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, s0 @@ -1180,6 +1243,9 @@ define amdgpu_kernel void @test_writelane_imm_i32(ptr addrspace(1) %out, i32 %sr ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s2, s[8:9], 0x8 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dword s3, s[0:1], 0x0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s0 @@ -1227,6 +1293,9 @@ define amdgpu_kernel void @test_writelane_imm_i64(ptr addrspace(1) %out, i64 %sr ; GFX802-SDAG-LABEL: test_writelane_imm_i64: ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 @@ -1270,6 +1339,9 @@ define amdgpu_kernel void @test_writelane_imm_i64(ptr addrspace(1) %out, i64 %sr ; GFX802-GISEL-LABEL: test_writelane_imm_i64: ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s1 @@ -1319,6 +1391,9 @@ define amdgpu_kernel void @test_writelane_imm_f64(ptr addrspace(1) %out, double ; GFX802-SDAG-LABEL: test_writelane_imm_f64: ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 @@ -1362,6 +1437,9 @@ define amdgpu_kernel void @test_writelane_imm_f64(ptr addrspace(1) %out, double ; GFX802-GISEL-LABEL: test_writelane_imm_f64: ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_load_dwordx2 s[4:5], s[0:1], 0x0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s1 @@ -1412,6 +1490,9 @@ define amdgpu_kernel void @test_writelane_sreg_oldval_i32(i32 inreg %oldval, ptr ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x8 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s4 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s3 @@ -1449,6 +1530,9 @@ define amdgpu_kernel void @test_writelane_sreg_oldval_i32(i32 inreg %oldval, ptr ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x8 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, s4 ; GFX802-GISEL-NEXT: s_mov_b32 m0, s3 @@ -1492,10 +1576,13 @@ define amdgpu_kernel void @test_writelane_sreg_oldval_i64(i64 inreg %oldval, ptr ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s6, s[8:9], 0x18 ; GFX802-SDAG-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s6 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s0 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, s2 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; GFX802-SDAG-NEXT: v_writelane_b32 v3, s5, m0 @@ -1538,11 +1625,14 @@ define amdgpu_kernel void @test_writelane_sreg_oldval_i64(i64 inreg %oldval, ptr ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s6, s[8:9], 0x18 ; GFX802-GISEL-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; GFX802-GISEL-NEXT: s_mov_b32 m0, s6 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, s2 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: v_writelane_b32 v0, s4, m0 ; GFX802-GISEL-NEXT: v_writelane_b32 v1, s5, m0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s3 @@ -1589,10 +1679,13 @@ define amdgpu_kernel void @test_writelane_sreg_oldval_f64(double inreg %oldval, ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s6, s[8:9], 0x18 ; GFX802-SDAG-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s6 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s0 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, s2 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, s3 ; GFX802-SDAG-NEXT: v_writelane_b32 v3, s5, m0 @@ -1635,11 +1728,14 @@ define amdgpu_kernel void @test_writelane_sreg_oldval_f64(double inreg %oldval, ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-GISEL-NEXT: s_load_dword s6, s[8:9], 0x18 ; GFX802-GISEL-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, s1 ; GFX802-GISEL-NEXT: s_mov_b32 m0, s6 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, s2 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: v_writelane_b32 v0, s4, m0 ; GFX802-GISEL-NEXT: v_writelane_b32 v1, s5, m0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s3 @@ -1684,7 +1780,10 @@ define amdgpu_kernel void @test_writelane_imm_oldval_i32(ptr addrspace(1) %out, ; GFX802-SDAG-LABEL: test_writelane_imm_oldval_i32: ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, 42 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-SDAG-NEXT: s_mov_b32 m0, s3 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, s0 @@ -1716,7 +1815,10 @@ define amdgpu_kernel void @test_writelane_imm_oldval_i32(ptr addrspace(1) %out, ; GFX802-GISEL-LABEL: test_writelane_imm_oldval_i32: ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, 42 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_mov_b32 m0, s3 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, s0 @@ -1754,11 +1856,14 @@ define amdgpu_kernel void @test_writelane_imm_oldval_i64(ptr addrspace(1) %out, ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, 0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, 42 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) -; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s4 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 ; GFX802-SDAG-NEXT: v_writelane_b32 v1, s3, m0 ; GFX802-SDAG-NEXT: v_writelane_b32 v0, s2, m0 @@ -1797,11 +1902,14 @@ define amdgpu_kernel void @test_writelane_imm_oldval_i64(ptr addrspace(1) %out, ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dword s4, s[8:9], 0x10 ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, 42 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, 0 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_mov_b32 m0, s4 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: v_writelane_b32 v0, s2, m0 ; GFX802-GISEL-NEXT: v_writelane_b32 v1, s3, m0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, s0 @@ -1845,11 +1953,14 @@ define amdgpu_kernel void @test_writelane_imm_oldval_f64(ptr addrspace(1) %out, ; GFX802-SDAG: ; %bb.0: ; GFX802-SDAG-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX802-SDAG-NEXT: s_load_dword s4, s[8:9], 0x10 +; GFX802-SDAG-NEXT: s_add_i32 s12, s12, s17 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v1, 0x40450000 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v0, 0 +; GFX802-SDAG-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX802-SDAG-NEXT: s_waitcnt lgkmcnt(0) -; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; GFX802-SDAG-NEXT: s_mov_b32 m0, s4 +; GFX802-SDAG-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX802-SDAG-NEXT: v_mov_b32_e32 v2, s0 ; GFX802-SDAG-NEXT: v_mov_b32_e32 v3, s1 ; GFX802-SDAG-NEXT: v_writelane_b32 v1, s3, m0 ; GFX802-SDAG-NEXT: v_writelane_b32 v0, s2, m0 @@ -1888,11 +1999,14 @@ define amdgpu_kernel void @test_writelane_imm_oldval_f64(ptr addrspace(1) %out, ; GFX802-GISEL: ; %bb.0: ; GFX802-GISEL-NEXT: s_load_dword s4, s[8:9], 0x10 ; GFX802-GISEL-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX802-GISEL-NEXT: s_add_i32 s12, s12, s17 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v0, 0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v1, 0x40450000 ; GFX802-GISEL-NEXT: s_waitcnt lgkmcnt(0) ; GFX802-GISEL-NEXT: s_mov_b32 m0, s4 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v3, s1 +; GFX802-GISEL-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX802-GISEL-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX802-GISEL-NEXT: v_writelane_b32 v0, s2, m0 ; GFX802-GISEL-NEXT: v_writelane_b32 v1, s3, m0 ; GFX802-GISEL-NEXT: v_mov_b32_e32 v2, s0 diff --git a/llvm/test/CodeGen/AMDGPU/load-constant-f64.ll b/llvm/test/CodeGen/AMDGPU/load-constant-f64.ll index 6f95364ac3644..919c1dfd4694e 100644 --- a/llvm/test/CodeGen/AMDGPU/load-constant-f64.ll +++ b/llvm/test/CodeGen/AMDGPU/load-constant-f64.ll @@ -22,6 +22,9 @@ define amdgpu_kernel void @constant_load_f64(ptr addrspace(1) %out, ptr addrspac ; GFX7-HSA-LABEL: constant_load_f64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -90,7 +93,10 @@ define amdgpu_kernel void @constant_load_2v4f64(ptr addrspace(4) noalias nocaptu ; ; GFX7-HSA-LABEL: constant_load_2v4f64: ; GFX7-HSA: ; %bb.0: ; %entry +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[20:21], s[18:19], 0x0 ; GFX7-HSA-NEXT: s_load_dwordx16 s[0:15], s[16:17], 0x0 diff --git a/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll b/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll index 51dfbda53ad4c..817c5def5614f 100644 --- a/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll +++ b/llvm/test/CodeGen/AMDGPU/load-constant-i16.ll @@ -27,6 +27,9 @@ define amdgpu_kernel void @constant_load_i16(ptr addrspace(1) %out, ptr addrspac ; GCN-HSA-LABEL: constant_load_i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -117,6 +120,9 @@ define amdgpu_kernel void @constant_load_v2i16(ptr addrspace(1) %out, ptr addrsp ; GCN-HSA-LABEL: constant_load_v2i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -188,6 +194,9 @@ define amdgpu_kernel void @constant_load_v3i16(ptr addrspace(1) %out, ptr addrsp ; GCN-HSA-LABEL: constant_load_v3i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GCN-HSA-NEXT: s_add_u32 s4, s0, 4 @@ -286,6 +295,9 @@ define amdgpu_kernel void @constant_load_v4i16(ptr addrspace(1) %out, ptr addrsp ; GCN-HSA-LABEL: constant_load_v4i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -360,6 +372,9 @@ define amdgpu_kernel void @constant_load_v8i16(ptr addrspace(1) %out, ptr addrsp ; GCN-HSA-LABEL: constant_load_v8i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -445,6 +460,9 @@ define amdgpu_kernel void @constant_load_v16i16(ptr addrspace(1) %out, ptr addrs ; GCN-HSA-LABEL: constant_load_v16i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 ; GCN-HSA-NEXT: s_add_u32 s10, s8, 16 @@ -584,6 +602,9 @@ define amdgpu_kernel void @constant_load_v16i16_align2(ptr addrspace(4) %ptr0) # ; GCN-HSA-LABEL: constant_load_v16i16_align2: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s0 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s1 @@ -831,6 +852,9 @@ define amdgpu_kernel void @constant_zextload_i16_to_i32(ptr addrspace(1) %out, p ; GCN-HSA-LABEL: constant_zextload_i16_to_i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -906,6 +930,9 @@ define amdgpu_kernel void @constant_sextload_i16_to_i32(ptr addrspace(1) %out, p ; GCN-HSA-LABEL: constant_sextload_i16_to_i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -982,6 +1009,9 @@ define amdgpu_kernel void @constant_zextload_v1i16_to_v1i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v1i16_to_v1i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1057,6 +1087,9 @@ define amdgpu_kernel void @constant_sextload_v1i16_to_v1i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v1i16_to_v1i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1131,6 +1164,9 @@ define amdgpu_kernel void @constant_zextload_v2i16_to_v2i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v2i16_to_v2i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -1216,6 +1252,9 @@ define amdgpu_kernel void @constant_sextload_v2i16_to_v2i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v2i16_to_v2i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -1305,6 +1344,9 @@ define amdgpu_kernel void @constant_zextload_v3i16_to_v3i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v3i16_to_v3i32: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v3, s0 @@ -1402,6 +1444,9 @@ define amdgpu_kernel void @constant_sextload_v3i16_to_v3i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v3i16_to_v3i32: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v3, s0 @@ -1504,6 +1549,9 @@ define amdgpu_kernel void @constant_zextload_v4i16_to_v4i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v4i16_to_v4i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -1610,6 +1658,9 @@ define amdgpu_kernel void @constant_sextload_v4i16_to_v4i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v4i16_to_v4i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -1727,6 +1778,9 @@ define amdgpu_kernel void @constant_zextload_v8i16_to_v8i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v8i16_to_v8i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -1885,6 +1939,9 @@ define amdgpu_kernel void @constant_sextload_v8i16_to_v8i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v8i16_to_v8i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2062,6 +2119,9 @@ define amdgpu_kernel void @constant_zextload_v16i16_to_v16i32(ptr addrspace(1) % ; GCN-HSA-LABEL: constant_zextload_v16i16_to_v16i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx8 s[4:11], s[2:3], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2324,6 +2384,9 @@ define amdgpu_kernel void @constant_sextload_v16i16_to_v16i32(ptr addrspace(1) % ; GCN-HSA-LABEL: constant_sextload_v16i16_to_v16i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx8 s[4:11], s[2:3], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2631,7 +2694,10 @@ define amdgpu_kernel void @constant_zextload_v32i16_to_v32i32(ptr addrspace(1) % ; ; GCN-HSA-LABEL: constant_zextload_v32i16_to_v32i32: ; GCN-HSA: ; %bb.0: +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 ; GCN-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -3112,7 +3178,10 @@ define amdgpu_kernel void @constant_sextload_v32i16_to_v32i32(ptr addrspace(1) % ; ; GCN-HSA-LABEL: constant_sextload_v32i16_to_v32i32: ; GCN-HSA: ; %bb.0: +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 ; GCN-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -3680,7 +3749,10 @@ define amdgpu_kernel void @constant_zextload_v64i16_to_v64i32(ptr addrspace(1) % ; ; GCN-HSA-LABEL: constant_zextload_v64i16_to_v64i32: ; GCN-HSA: ; %bb.0: +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 ; GCN-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -4596,7 +4668,10 @@ define amdgpu_kernel void @constant_sextload_v64i16_to_v64i32(ptr addrspace(1) % ; ; GCN-HSA-LABEL: constant_sextload_v64i16_to_v64i32: ; GCN-HSA: ; %bb.0: +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 ; GCN-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -5383,6 +5458,9 @@ define amdgpu_kernel void @constant_zextload_i16_to_i64(ptr addrspace(1) %out, p ; GCN-HSA-LABEL: constant_zextload_i16_to_i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5480,6 +5558,9 @@ define amdgpu_kernel void @constant_sextload_i16_to_i64(ptr addrspace(1) %out, p ; GCN-HSA-LABEL: constant_sextload_i16_to_i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5578,6 +5659,9 @@ define amdgpu_kernel void @constant_zextload_v1i16_to_v1i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v1i16_to_v1i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5670,6 +5754,9 @@ define amdgpu_kernel void @constant_sextload_v1i16_to_v1i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v1i16_to_v1i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5767,12 +5854,15 @@ define amdgpu_kernel void @constant_zextload_v2i16_to_v2i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v2i16_to_v2i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v4, s0 ; GCN-HSA-NEXT: v_mov_b32_e32 v5, s1 +; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_lshr_b32 s0, s2, 16 ; GCN-HSA-NEXT: s_and_b32 s1, s2, 0xffff @@ -5877,6 +5967,9 @@ define amdgpu_kernel void @constant_sextload_v2i16_to_v2i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v2i16_to_v2i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GCN-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -5980,10 +6073,13 @@ define amdgpu_kernel void @constant_zextload_v4i16_to_v4i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v4i16_to_v4i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 +; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_lshr_b32 s4, s3, 16 ; GCN-HSA-NEXT: s_lshr_b32 s5, s2, 16 @@ -6136,6 +6232,9 @@ define amdgpu_kernel void @constant_sextload_v4i16_to_v4i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v4i16_to_v4i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -6292,10 +6391,13 @@ define amdgpu_kernel void @constant_zextload_v8i16_to_v8i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_zextload_v8i16_to_v8i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 +; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_lshr_b32 s8, s5, 16 ; GCN-HSA-NEXT: s_lshr_b32 s2, s7, 16 @@ -6510,6 +6612,9 @@ define amdgpu_kernel void @constant_sextload_v8i16_to_v8i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: constant_sextload_v8i16_to_v8i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -6771,10 +6876,13 @@ define amdgpu_kernel void @constant_zextload_v16i16_to_v16i64(ptr addrspace(1) % ; GCN-HSA-LABEL: constant_zextload_v16i16_to_v16i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx8 s[4:11], s[2:3], 0x0 +; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_lshr_b32 s12, s5, 16 ; GCN-HSA-NEXT: s_lshr_b32 s13, s7, 16 @@ -7156,6 +7264,9 @@ define amdgpu_kernel void @constant_sextload_v16i16_to_v16i64(ptr addrspace(1) % ; GCN-HSA-LABEL: constant_sextload_v16i16_to_v16i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx8 s[12:19], s[2:3], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -7631,7 +7742,10 @@ define amdgpu_kernel void @constant_zextload_v32i16_to_v32i64(ptr addrspace(1) % ; ; GCN-HSA-LABEL: constant_zextload_v32i16_to_v32i64: ; GCN-HSA: ; %bb.0: +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 ; GCN-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, 0 ; GCN-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -8354,7 +8468,10 @@ define amdgpu_kernel void @constant_sextload_v32i16_to_v32i64(ptr addrspace(1) % ; ; GCN-HSA-LABEL: constant_sextload_v32i16_to_v32i64: ; GCN-HSA: ; %bb.0: +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 ; GCN-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/load-constant-i32.ll b/llvm/test/CodeGen/AMDGPU/load-constant-i32.ll index 120f47a277ee6..68a6a148819e8 100644 --- a/llvm/test/CodeGen/AMDGPU/load-constant-i32.ll +++ b/llvm/test/CodeGen/AMDGPU/load-constant-i32.ll @@ -23,6 +23,9 @@ define amdgpu_kernel void @constant_load_i32(ptr addrspace(1) %out, ptr addrspac ; GFX7-HSA-LABEL: constant_load_i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -103,6 +106,9 @@ define amdgpu_kernel void @constant_load_v2i32(ptr addrspace(1) %out, ptr addrsp ; GFX7-HSA-LABEL: constant_load_v2i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -190,6 +196,9 @@ define amdgpu_kernel void @constant_load_v3i32(ptr addrspace(1) %out, ptr addrsp ; GFX7-HSA-LABEL: constant_load_v3i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v3, s0 @@ -284,6 +293,9 @@ define amdgpu_kernel void @constant_load_v4i32(ptr addrspace(1) %out, ptr addrsp ; GFX7-HSA-LABEL: constant_load_v4i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -383,6 +395,9 @@ define amdgpu_kernel void @constant_load_v8i32(ptr addrspace(1) %out, ptr addrsp ; GFX7-HSA-LABEL: constant_load_v8i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 ; GFX7-HSA-NEXT: s_add_u32 s10, s8, 16 @@ -517,6 +532,9 @@ define amdgpu_kernel void @constant_load_v9i32(ptr addrspace(1) %out, ptr addrsp ; GFX7-HSA-LABEL: constant_load_v9i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s12, s[10:11], 0x8 ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 @@ -678,6 +696,9 @@ define amdgpu_kernel void @constant_load_v10i32(ptr addrspace(1) %out, ptr addrs ; GFX7-HSA-LABEL: constant_load_v10i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[12:13], s[10:11], 0x8 ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 @@ -847,6 +868,9 @@ define amdgpu_kernel void @constant_load_v11i32(ptr addrspace(1) %out, ptr addrs ; GFX7-HSA-LABEL: constant_load_v11i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[12:15], s[10:11], 0x8 ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 @@ -1023,6 +1047,9 @@ define amdgpu_kernel void @constant_load_v12i32(ptr addrspace(1) %out, ptr addrs ; GFX7-HSA-LABEL: constant_load_v12i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[12:15], s[10:11], 0x8 ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 @@ -1202,7 +1229,10 @@ define amdgpu_kernel void @constant_load_v16i32(ptr addrspace(1) %out, ptr addrs ; ; GFX7-HSA-LABEL: constant_load_v16i32: ; GFX7-HSA: ; %bb.0: ; %entry +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GFX7-HSA-NEXT: s_add_u32 s18, s16, 48 @@ -1389,6 +1419,9 @@ define amdgpu_kernel void @constant_zextload_i32_to_i64(ptr addrspace(1) %out, p ; GFX7-HSA-LABEL: constant_zextload_i32_to_i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v3, 0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 @@ -1473,6 +1506,9 @@ define amdgpu_kernel void @constant_sextload_i32_to_i64(ptr addrspace(1) %out, p ; GFX7-HSA-LABEL: constant_sextload_i32_to_i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -1563,6 +1599,9 @@ define amdgpu_kernel void @constant_zextload_v1i32_to_v1i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_zextload_v1i32_to_v1i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v3, 0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 @@ -1647,6 +1686,9 @@ define amdgpu_kernel void @constant_sextload_v1i32_to_v1i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_sextload_v1i32_to_v1i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -1739,12 +1781,15 @@ define amdgpu_kernel void @constant_zextload_v2i32_to_v2i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_zextload_v2i32_to_v2i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v5, s1 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v2, s3 @@ -1837,6 +1882,9 @@ define amdgpu_kernel void @constant_sextload_v2i32_to_v2i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_sextload_v2i32_to_v2i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -1949,13 +1997,16 @@ define amdgpu_kernel void @constant_zextload_v4i32_to_v4i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_zextload_v4i32_to_v4i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_add_u32 s2, s0, 16 ; GFX7-HSA-NEXT: s_addc_u32 s3, s1, 0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v5, s3 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s2 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s6 @@ -2082,6 +2133,9 @@ define amdgpu_kernel void @constant_sextload_v4i32_to_v4i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_sextload_v4i32_to_v4i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2244,8 +2298,10 @@ define amdgpu_kernel void @constant_zextload_v8i32_to_v8i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_zextload_v8i32_to_v8i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[4:11], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_add_u32 s2, s0, 48 @@ -2253,6 +2309,7 @@ define amdgpu_kernel void @constant_zextload_v8i32_to_v8i64(ptr addrspace(1) %ou ; GFX7-HSA-NEXT: v_mov_b32_e32 v5, s3 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s2 ; GFX7-HSA-NEXT: s_add_u32 s2, s0, 32 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s10 ; GFX7-HSA-NEXT: v_mov_b32_e32 v2, s11 @@ -2452,6 +2509,9 @@ define amdgpu_kernel void @constant_sextload_v8i32_to_v8i64(ptr addrspace(1) %ou ; GFX7-HSA-LABEL: constant_sextload_v8i32_to_v8i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2748,7 +2808,10 @@ define amdgpu_kernel void @constant_sextload_v16i32_to_v16i64(ptr addrspace(1) % ; ; GFX7-HSA-LABEL: constant_sextload_v16i32_to_v16i64: ; GFX7-HSA: ; %bb.0: +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -3196,7 +3259,10 @@ define amdgpu_kernel void @constant_zextload_v16i32_to_v16i64(ptr addrspace(1) % ; ; GFX7-HSA-LABEL: constant_zextload_v16i32_to_v16i64: ; GFX7-HSA: ; %bb.0: +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -3628,7 +3694,10 @@ define amdgpu_kernel void @constant_sextload_v32i32_to_v32i64(ptr addrspace(1) % ; ; GFX7-HSA-LABEL: constant_sextload_v32i32_to_v32i64: ; GFX7-HSA: ; %bb.0: +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x10 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -4479,8 +4548,10 @@ define amdgpu_kernel void @constant_zextload_v32i32_to_v32i64(ptr addrspace(1) % ; GFX7-HSA-LABEL: constant_zextload_v32i32_to_v32i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[36:39], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_add_u32 s0, s36, 0xf0 ; GFX7-HSA-NEXT: s_addc_u32 s1, s37, 0 @@ -4509,6 +4580,7 @@ define amdgpu_kernel void @constant_zextload_v32i32_to_v32i64(ptr addrspace(1) % ; GFX7-HSA-NEXT: v_mov_b32_e32 v14, s0 ; GFX7-HSA-NEXT: s_add_u32 s0, s36, 0x90 ; GFX7-HSA-NEXT: s_addc_u32 s1, s37, 0 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s30 ; GFX7-HSA-NEXT: v_mov_b32_e32 v2, s31 @@ -5097,7 +5169,10 @@ define amdgpu_kernel void @constant_load_v32i32(ptr addrspace(1) %out, ptr addrs ; ; GFX7-HSA-LABEL: constant_load_v32i32: ; GFX7-HSA: ; %bb.0: +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x10 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/load-constant-i64.ll b/llvm/test/CodeGen/AMDGPU/load-constant-i64.ll index b3e75e767ae64..2219ceea7ec9b 100644 --- a/llvm/test/CodeGen/AMDGPU/load-constant-i64.ll +++ b/llvm/test/CodeGen/AMDGPU/load-constant-i64.ll @@ -22,6 +22,9 @@ define amdgpu_kernel void @constant_load_i64(ptr addrspace(1) %out, ptr addrspac ; GFX7-LABEL: constant_load_i64: ; GFX7: ; %bb.0: ; GFX7-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-NEXT: v_mov_b32_e32 v0, s0 @@ -95,6 +98,9 @@ define amdgpu_kernel void @constant_load_v2i64(ptr addrspace(1) %out, ptr addrsp ; GFX7-LABEL: constant_load_v2i64: ; GFX7: ; %bb.0: ; %entry ; GFX7-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-NEXT: v_mov_b32_e32 v4, s0 @@ -179,6 +185,9 @@ define amdgpu_kernel void @constant_load_v3i64(ptr addrspace(1) %out, ptr addrsp ; GFX7-LABEL: constant_load_v3i64: ; GFX7: ; %bb.0: ; %entry ; GFX7-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[2:3], 0x4 ; GFX7-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 @@ -294,6 +303,9 @@ define amdgpu_kernel void @constant_load_v4i64(ptr addrspace(1) %out, ptr addrsp ; GFX7-LABEL: constant_load_v4i64: ; GFX7: ; %bb.0: ; %entry ; GFX7-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 ; GFX7-NEXT: s_add_u32 s10, s8, 16 @@ -421,7 +433,10 @@ define amdgpu_kernel void @constant_load_v8i64(ptr addrspace(1) %out, ptr addrsp ; ; GFX7-LABEL: constant_load_v8i64: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_add_i32 s12, s12, s17 ; GFX7-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GFX7-NEXT: s_add_u32 s18, s16, 48 @@ -638,7 +653,10 @@ define amdgpu_kernel void @constant_load_v16i64(ptr addrspace(1) %out, ptr addrs ; ; GFX7-LABEL: constant_load_v16i64: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_add_i32 s12, s12, s17 ; GFX7-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x10 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/load-constant-i8.ll b/llvm/test/CodeGen/AMDGPU/load-constant-i8.ll index c608bef3f726e..4031be65fab61 100644 --- a/llvm/test/CodeGen/AMDGPU/load-constant-i8.ll +++ b/llvm/test/CodeGen/AMDGPU/load-constant-i8.ll @@ -27,6 +27,9 @@ define amdgpu_kernel void @constant_load_i8(ptr addrspace(1) %out, ptr addrspace ; GFX7-HSA-LABEL: constant_load_i8: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -112,6 +115,9 @@ define amdgpu_kernel void @constant_load_v2i8(ptr addrspace(1) %out, ptr addrspa ; GFX7-HSA-LABEL: constant_load_v2i8: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -195,6 +201,9 @@ define amdgpu_kernel void @constant_load_v3i8(ptr addrspace(1) %out, ptr addrspa ; GFX7-HSA-LABEL: constant_load_v3i8: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -305,6 +314,9 @@ define amdgpu_kernel void @constant_load_v4i8(ptr addrspace(1) %out, ptr addrspa ; GFX7-HSA-LABEL: constant_load_v4i8: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -374,6 +386,9 @@ define amdgpu_kernel void @constant_load_v8i8(ptr addrspace(1) %out, ptr addrspa ; GFX7-HSA-LABEL: constant_load_v8i8: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -448,6 +463,9 @@ define amdgpu_kernel void @constant_load_v16i8(ptr addrspace(1) %out, ptr addrsp ; GFX7-HSA-LABEL: constant_load_v16i8: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -529,6 +547,9 @@ define amdgpu_kernel void @constant_zextload_i8_to_i32(ptr addrspace(1) %out, pt ; GFX7-HSA-LABEL: constant_zextload_i8_to_i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -604,6 +625,9 @@ define amdgpu_kernel void @constant_sextload_i8_to_i32(ptr addrspace(1) %out, pt ; GFX7-HSA-LABEL: constant_sextload_i8_to_i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -680,6 +704,9 @@ define amdgpu_kernel void @constant_zextload_v1i8_to_v1i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v1i8_to_v1i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -755,6 +782,9 @@ define amdgpu_kernel void @constant_sextload_v1i8_to_v1i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v1i8_to_v1i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -834,6 +864,9 @@ define amdgpu_kernel void @constant_zextload_v2i8_to_v2i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v2i8_to_v2i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -933,6 +966,9 @@ define amdgpu_kernel void @constant_sextload_v2i8_to_v2i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v2i8_to_v2i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1030,6 +1066,9 @@ define amdgpu_kernel void @constant_zextload_v3i8_to_v3i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v3i8_to_v3i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v3, s0 @@ -1131,6 +1170,9 @@ define amdgpu_kernel void @constant_sextload_v3i8_to_v3i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v3i8_to_v3i32: ; GFX7-HSA: ; %bb.0: ; %entry ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v3, s0 @@ -1232,6 +1274,9 @@ define amdgpu_kernel void @constant_zextload_v4i8_to_v4i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v4i8_to_v4i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -1336,6 +1381,9 @@ define amdgpu_kernel void @constant_sextload_v4i8_to_v4i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v4i8_to_v4i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -1453,6 +1501,9 @@ define amdgpu_kernel void @constant_zextload_v8i8_to_v8i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v8i8_to_v8i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -1612,6 +1663,9 @@ define amdgpu_kernel void @constant_sextload_v8i8_to_v8i32(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v8i8_to_v8i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -1794,6 +1848,9 @@ define amdgpu_kernel void @constant_zextload_v16i8_to_v16i32(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_zextload_v16i8_to_v16i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2060,6 +2117,9 @@ define amdgpu_kernel void @constant_sextload_v16i8_to_v16i32(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_sextload_v16i8_to_v16i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2374,6 +2434,9 @@ define amdgpu_kernel void @constant_zextload_v32i8_to_v32i32(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_zextload_v32i8_to_v32i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[4:11], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -2856,6 +2919,9 @@ define amdgpu_kernel void @constant_sextload_v32i8_to_v32i32(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_sextload_v32i8_to_v32i32: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[4:11], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -3437,7 +3503,10 @@ define amdgpu_kernel void @constant_zextload_v64i8_to_v64i32(ptr addrspace(1) %o ; ; GFX7-HSA-LABEL: constant_zextload_v64i8_to_v64i32: ; GFX7-HSA: ; %bb.0: +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -4353,7 +4422,10 @@ define amdgpu_kernel void @constant_sextload_v64i8_to_v64i32(ptr addrspace(1) %o ; ; GFX7-HSA-LABEL: constant_sextload_v64i8_to_v64i32: ; GFX7-HSA: ; %bb.0: +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 ; GFX7-HSA-NEXT: s_load_dwordx4 s[16:19], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx16 s[0:15], s[18:19], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -5161,6 +5233,9 @@ define amdgpu_kernel void @constant_zextload_i8_to_i64(ptr addrspace(1) %out, pt ; GFX7-HSA-LABEL: constant_zextload_i8_to_i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5243,6 +5318,9 @@ define amdgpu_kernel void @constant_sextload_i8_to_i64(ptr addrspace(1) %out, pt ; GFX7-HSA-LABEL: constant_sextload_i8_to_i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5328,6 +5406,9 @@ define amdgpu_kernel void @constant_zextload_v1i8_to_v1i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v1i8_to_v1i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5408,6 +5489,9 @@ define amdgpu_kernel void @constant_sextload_v1i8_to_v1i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v1i8_to_v1i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5496,6 +5580,9 @@ define amdgpu_kernel void @constant_zextload_v2i8_to_v2i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v2i8_to_v2i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5603,6 +5690,9 @@ define amdgpu_kernel void @constant_sextload_v2i8_to_v2i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v2i8_to_v2i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5716,10 +5806,13 @@ define amdgpu_kernel void @constant_zextload_v4i8_to_v4i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v4i8_to_v4i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_bfe_u32 s4, s2, 0x80008 ; GFX7-HSA-NEXT: s_lshr_b32 s3, s2, 24 @@ -5854,6 +5947,9 @@ define amdgpu_kernel void @constant_sextload_v4i8_to_v4i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v4i8_to_v4i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -6013,10 +6109,13 @@ define amdgpu_kernel void @constant_zextload_v8i8_to_v8i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v8i8_to_v8i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_lshr_b32 s4, s2, 24 ; GFX7-HSA-NEXT: s_lshr_b32 s5, s3, 24 @@ -6235,6 +6334,9 @@ define amdgpu_kernel void @constant_sextload_v8i8_to_v8i64(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v8i8_to_v8i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -6504,10 +6606,13 @@ define amdgpu_kernel void @constant_zextload_v16i8_to_v16i64(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_zextload_v16i8_to_v16i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_lshr_b32 s8, s5, 24 ; GFX7-HSA-NEXT: s_lshr_b32 s9, s4, 24 @@ -6898,6 +7003,9 @@ define amdgpu_kernel void @constant_sextload_v16i8_to_v16i64(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_sextload_v16i8_to_v16i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -7387,10 +7495,13 @@ define amdgpu_kernel void @constant_zextload_v32i8_to_v32i64(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_zextload_v32i8_to_v32i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, 0 -; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[8:15], s[2:3], 0x0 +; GFX7-HSA-NEXT: v_mov_b32_e32 v3, v1 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_lshr_b32 s16, s8, 24 ; GFX7-HSA-NEXT: s_lshr_b32 s17, s9, 24 @@ -8128,6 +8239,9 @@ define amdgpu_kernel void @constant_sextload_v32i8_to_v32i64(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_sextload_v32i8_to_v32i64: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -8898,6 +9012,9 @@ define amdgpu_kernel void @constant_zextload_i8_to_i16(ptr addrspace(1) %out, pt ; GFX7-HSA-LABEL: constant_zextload_i8_to_i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -8982,6 +9099,9 @@ define amdgpu_kernel void @constant_sextload_i8_to_i16(ptr addrspace(1) %out, pt ; GFX7-HSA-LABEL: constant_sextload_i8_to_i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -9068,6 +9188,9 @@ define amdgpu_kernel void @constant_zextload_v1i8_to_v1i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v1i8_to_v1i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -9152,6 +9275,9 @@ define amdgpu_kernel void @constant_sextload_v1i8_to_v1i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v1i8_to_v1i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -9241,6 +9367,9 @@ define amdgpu_kernel void @constant_zextload_v2i8_to_v2i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v2i8_to_v2i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -9340,6 +9469,9 @@ define amdgpu_kernel void @constant_sextload_v2i8_to_v2i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v2i8_to_v2i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GFX7-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -9452,6 +9584,9 @@ define amdgpu_kernel void @constant_zextload_v4i8_to_v4i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v4i8_to_v4i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -9560,6 +9695,9 @@ define amdgpu_kernel void @constant_sextload_v4i8_to_v4i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v4i8_to_v4i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dword s2, s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v0, s0 @@ -9683,6 +9821,9 @@ define amdgpu_kernel void @constant_zextload_v8i8_to_v8i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_zextload_v8i8_to_v8i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -9832,6 +9973,9 @@ define amdgpu_kernel void @constant_sextload_v8i8_to_v8i16(ptr addrspace(1) %out ; GFX7-HSA-LABEL: constant_sextload_v8i8_to_v8i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx2 s[2:3], s[2:3], 0x0 ; GFX7-HSA-NEXT: v_mov_b32_e32 v4, s0 @@ -10014,6 +10158,9 @@ define amdgpu_kernel void @constant_zextload_v16i8_to_v16i16(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_zextload_v16i8_to_v16i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -10261,6 +10408,9 @@ define amdgpu_kernel void @constant_sextload_v16i8_to_v16i16(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_sextload_v16i8_to_v16i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx4 s[4:7], s[2:3], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -10574,6 +10724,9 @@ define amdgpu_kernel void @constant_zextload_v32i8_to_v32i16(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_zextload_v32i8_to_v32i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) @@ -11018,6 +11171,9 @@ define amdgpu_kernel void @constant_sextload_v32i8_to_v32i16(ptr addrspace(1) %o ; GFX7-HSA-LABEL: constant_sextload_v32i8_to_v32i16: ; GFX7-HSA: ; %bb.0: ; GFX7-HSA-NEXT: s_load_dwordx4 s[8:11], s[8:9], 0x0 +; GFX7-HSA-NEXT: s_add_i32 s12, s12, s17 +; GFX7-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GFX7-HSA-NEXT: s_load_dwordx8 s[0:7], s[10:11], 0x0 ; GFX7-HSA-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/load-global-i16.ll b/llvm/test/CodeGen/AMDGPU/load-global-i16.ll index c5771bc73b945..9054e509cde8e 100644 --- a/llvm/test/CodeGen/AMDGPU/load-global-i16.ll +++ b/llvm/test/CodeGen/AMDGPU/load-global-i16.ll @@ -28,6 +28,9 @@ define amdgpu_kernel void @global_load_i16(ptr addrspace(1) %out, ptr addrspace( ; GCN-HSA-LABEL: global_load_i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -133,6 +136,9 @@ define amdgpu_kernel void @global_load_v2i16(ptr addrspace(1) %out, ptr addrspac ; GCN-HSA-LABEL: global_load_v2i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -219,6 +225,9 @@ define amdgpu_kernel void @global_load_v3i16(ptr addrspace(1) %out, ptr addrspac ; GCN-HSA-LABEL: global_load_v3i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -339,6 +348,9 @@ define amdgpu_kernel void @global_load_v4i16(ptr addrspace(1) %out, ptr addrspac ; GCN-HSA-LABEL: global_load_v4i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -424,6 +436,9 @@ define amdgpu_kernel void @global_load_v8i16(ptr addrspace(1) %out, ptr addrspac ; GCN-HSA-LABEL: global_load_v8i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -512,6 +527,9 @@ define amdgpu_kernel void @global_load_v16i16(ptr addrspace(1) %out, ptr addrspa ; GCN-HSA-LABEL: global_load_v16i16: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_add_u32 s4, s0, 16 ; GCN-HSA-NEXT: v_mov_b32_e32 v5, s3 @@ -662,6 +680,9 @@ define amdgpu_kernel void @global_load_v16i16_align2(ptr addrspace(1) %in, ptr a ; GCN-HSA-LABEL: global_load_v16i16_align2: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCN-HSA-NEXT: v_mov_b32_e32 v5, s1 @@ -811,6 +832,9 @@ define amdgpu_kernel void @global_zextload_i16_to_i32(ptr addrspace(1) %out, ptr ; GCN-HSA-LABEL: global_zextload_i16_to_i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -896,6 +920,9 @@ define amdgpu_kernel void @global_sextload_i16_to_i32(ptr addrspace(1) %out, ptr ; GCN-HSA-LABEL: global_sextload_i16_to_i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -984,6 +1011,9 @@ define amdgpu_kernel void @global_zextload_v1i16_to_v1i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v1i16_to_v1i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1069,6 +1099,9 @@ define amdgpu_kernel void @global_sextload_v1i16_to_v1i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v1i16_to_v1i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1159,6 +1192,9 @@ define amdgpu_kernel void @global_zextload_v2i16_to_v2i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v2i16_to_v2i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1258,6 +1294,9 @@ define amdgpu_kernel void @global_sextload_v2i16_to_v2i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v2i16_to_v2i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1359,6 +1398,9 @@ define amdgpu_kernel void @global_zextload_v3i16_to_v3i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v3i16_to_v3i32: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1469,6 +1511,9 @@ define amdgpu_kernel void @global_sextload_v3i16_to_v3i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v3i16_to_v3i32: ; GCN-HSA: ; %bb.0: ; %entry ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1586,6 +1631,9 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v4i16_to_v4i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1701,6 +1749,9 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v4i16_to_v4i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1823,6 +1874,9 @@ define amdgpu_kernel void @global_zextload_v8i16_to_v8i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v8i16_to_v8i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1972,6 +2026,9 @@ define amdgpu_kernel void @global_sextload_v8i16_to_v8i32(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v8i16_to_v8i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -2136,6 +2193,9 @@ define amdgpu_kernel void @global_zextload_v16i16_to_v16i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_zextload_v16i16_to_v16i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCN-HSA-NEXT: s_addc_u32 s5, s3, 0 @@ -2372,6 +2432,9 @@ define amdgpu_kernel void @global_sextload_v16i16_to_v16i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_sextload_v16i16_to_v16i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -2643,6 +2706,9 @@ define amdgpu_kernel void @global_zextload_v32i16_to_v32i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_zextload_v32i16_to_v32i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCN-HSA-NEXT: s_addc_u32 s5, s3, 0 @@ -3054,6 +3120,9 @@ define amdgpu_kernel void @global_sextload_v32i16_to_v32i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_sextload_v32i16_to_v32i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -3573,6 +3642,9 @@ define amdgpu_kernel void @global_zextload_v64i16_to_v64i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_zextload_v64i16_to_v64i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -4377,6 +4449,9 @@ define amdgpu_kernel void @global_sextload_v64i16_to_v64i32(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_sextload_v64i16_to_v64i32: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5142,6 +5217,9 @@ define amdgpu_kernel void @global_zextload_i16_to_i64(ptr addrspace(1) %out, ptr ; GCN-HSA-LABEL: global_zextload_i16_to_i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5239,6 +5317,9 @@ define amdgpu_kernel void @global_sextload_i16_to_i64(ptr addrspace(1) %out, ptr ; GCN-HSA-LABEL: global_sextload_i16_to_i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5334,6 +5415,9 @@ define amdgpu_kernel void @global_zextload_v1i16_to_v1i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v1i16_to_v1i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5426,6 +5510,9 @@ define amdgpu_kernel void @global_sextload_v1i16_to_v1i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v1i16_to_v1i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5524,6 +5611,9 @@ define amdgpu_kernel void @global_zextload_v2i16_to_v2i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v2i16_to_v2i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5633,6 +5723,9 @@ define amdgpu_kernel void @global_sextload_v2i16_to_v2i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v2i16_to_v2i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5751,6 +5844,9 @@ define amdgpu_kernel void @global_zextload_v4i16_to_v4i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v4i16_to_v4i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -5896,6 +5992,9 @@ define amdgpu_kernel void @global_sextload_v4i16_to_v4i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v4i16_to_v4i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -6056,10 +6155,10 @@ define amdgpu_kernel void @global_zextload_v8i16_to_v8i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_zextload_v8i16_to_v8i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: v_mov_b32_e32 v4, 0 -; GCN-HSA-NEXT: v_mov_b32_e32 v6, v4 -; GCN-HSA-NEXT: v_mov_b32_e32 v8, v4 -; GCN-HSA-NEXT: v_mov_b32_e32 v10, v4 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -6074,8 +6173,11 @@ define amdgpu_kernel void @global_zextload_v8i16_to_v8i64(ptr addrspace(1) %out, ; GCN-HSA-NEXT: v_mov_b32_e32 v19, s0 ; GCN-HSA-NEXT: s_add_u32 s0, s0, 32 ; GCN-HSA-NEXT: s_addc_u32 s1, s1, 0 +; GCN-HSA-NEXT: v_mov_b32_e32 v6, v4 ; GCN-HSA-NEXT: v_mov_b32_e32 v24, s3 ; GCN-HSA-NEXT: v_mov_b32_e32 v26, s1 +; GCN-HSA-NEXT: v_mov_b32_e32 v8, v4 +; GCN-HSA-NEXT: v_mov_b32_e32 v10, v4 ; GCN-HSA-NEXT: v_mov_b32_e32 v12, v4 ; GCN-HSA-NEXT: v_mov_b32_e32 v14, v4 ; GCN-HSA-NEXT: v_mov_b32_e32 v16, v4 @@ -6275,6 +6377,9 @@ define amdgpu_kernel void @global_sextload_v8i16_to_v8i64(ptr addrspace(1) %out, ; GCN-HSA-LABEL: global_sextload_v8i16_to_v8i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -6525,10 +6630,10 @@ define amdgpu_kernel void @global_zextload_v16i16_to_v16i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_zextload_v16i16_to_v16i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: v_mov_b32_e32 v8, 0 -; GCN-HSA-NEXT: v_mov_b32_e32 v12, v8 -; GCN-HSA-NEXT: v_mov_b32_e32 v14, v8 -; GCN-HSA-NEXT: v_mov_b32_e32 v15, v8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -6545,7 +6650,10 @@ define amdgpu_kernel void @global_zextload_v16i16_to_v16i64(ptr addrspace(1) %ou ; GCN-HSA-NEXT: v_mov_b32_e32 v22, s5 ; GCN-HSA-NEXT: v_mov_b32_e32 v21, s4 ; GCN-HSA-NEXT: s_add_u32 s4, s0, 0x50 +; GCN-HSA-NEXT: v_mov_b32_e32 v12, v8 +; GCN-HSA-NEXT: v_mov_b32_e32 v14, v8 ; GCN-HSA-NEXT: s_addc_u32 s5, s1, 0 +; GCN-HSA-NEXT: v_mov_b32_e32 v15, v8 ; GCN-HSA-NEXT: v_mov_b32_e32 v17, v8 ; GCN-HSA-NEXT: v_mov_b32_e32 v18, v8 ; GCN-HSA-NEXT: v_mov_b32_e32 v20, v8 @@ -6905,6 +7013,9 @@ define amdgpu_kernel void @global_sextload_v16i16_to_v16i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_sextload_v16i16_to_v16i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -7376,6 +7487,9 @@ define amdgpu_kernel void @global_zextload_v32i16_to_v32i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_zextload_v32i16_to_v32i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCN-HSA-NEXT: s_addc_u32 s5, s3, 0 @@ -8078,6 +8192,9 @@ define amdgpu_kernel void @global_sextload_v32i16_to_v32i64(ptr addrspace(1) %ou ; GCN-HSA-LABEL: global_sextload_v32i16_to_v32i64: ; GCN-HSA: ; %bb.0: ; GCN-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCN-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCN-HSA-NEXT: v_mov_b32_e32 v1, s3 diff --git a/llvm/test/CodeGen/AMDGPU/load-global-i32.ll b/llvm/test/CodeGen/AMDGPU/load-global-i32.ll index 033a66abcedb9..e8c862a3cb93c 100644 --- a/llvm/test/CodeGen/AMDGPU/load-global-i32.ll +++ b/llvm/test/CodeGen/AMDGPU/load-global-i32.ll @@ -27,6 +27,9 @@ define amdgpu_kernel void @global_load_i32(ptr addrspace(1) %out, ptr addrspace( ; GCNX3-HSA-LABEL: global_load_i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -106,6 +109,9 @@ define amdgpu_kernel void @global_load_v2i32(ptr addrspace(1) %out, ptr addrspac ; GCNX3-HSA-LABEL: global_load_v2i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -186,6 +192,9 @@ define amdgpu_kernel void @global_load_v3i32(ptr addrspace(1) %out, ptr addrspac ; GCNX3-HSA-LABEL: global_load_v3i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -270,6 +279,9 @@ define amdgpu_kernel void @global_load_v4i32(ptr addrspace(1) %out, ptr addrspac ; GCNX3-HSA-LABEL: global_load_v4i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -352,6 +364,9 @@ define amdgpu_kernel void @global_load_v8i32(ptr addrspace(1) %out, ptr addrspac ; GCNX3-HSA-LABEL: global_load_v8i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v5, s3 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v4, s2 @@ -458,6 +473,9 @@ define amdgpu_kernel void @global_load_v9i32(ptr addrspace(1) %out, ptr addrspac ; GCNX3-HSA-LABEL: global_load_v9i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 32 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 @@ -589,6 +607,9 @@ define amdgpu_kernel void @global_load_v10i32(ptr addrspace(1) %out, ptr addrspa ; GCNX3-HSA-LABEL: global_load_v10i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 32 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 @@ -719,6 +740,9 @@ define amdgpu_kernel void @global_load_v11i32(ptr addrspace(1) %out, ptr addrspa ; GCNX3-HSA-LABEL: global_load_v11i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 32 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 @@ -854,6 +878,9 @@ define amdgpu_kernel void @global_load_v12i32(ptr addrspace(1) %out, ptr addrspa ; GCNX3-HSA-LABEL: global_load_v12i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 32 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 @@ -987,6 +1014,9 @@ define amdgpu_kernel void @global_load_v16i32(ptr addrspace(1) %out, ptr addrspa ; GCNX3-HSA-LABEL: global_load_v16i32: ; GCNX3-HSA: ; %bb.0: ; %entry ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCNX3-HSA-NEXT: s_addc_u32 s5, s3, 0 @@ -1134,6 +1164,9 @@ define amdgpu_kernel void @global_zextload_i32_to_i64(ptr addrspace(1) %out, ptr ; GCNX3-HSA-LABEL: global_zextload_i32_to_i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1217,6 +1250,9 @@ define amdgpu_kernel void @global_sextload_i32_to_i64(ptr addrspace(1) %out, ptr ; GCNX3-HSA-LABEL: global_sextload_i32_to_i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1301,6 +1337,9 @@ define amdgpu_kernel void @global_zextload_v1i32_to_v1i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_zextload_v1i32_to_v1i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1384,6 +1423,9 @@ define amdgpu_kernel void @global_sextload_v1i32_to_v1i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_sextload_v1i32_to_v1i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1471,6 +1513,9 @@ define amdgpu_kernel void @global_zextload_v2i32_to_v2i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_zextload_v2i32_to_v2i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1569,6 +1614,9 @@ define amdgpu_kernel void @global_sextload_v2i32_to_v2i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_sextload_v2i32_to_v2i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1674,8 +1722,10 @@ define amdgpu_kernel void @global_zextload_v4i32_to_v4i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_zextload_v4i32_to_v4i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v5, 0 -; GCNX3-HSA-NEXT: v_mov_b32_e32 v7, v5 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1683,6 +1733,7 @@ define amdgpu_kernel void @global_zextload_v4i32_to_v4i64(ptr addrspace(1) %out, ; GCNX3-HSA-NEXT: s_add_u32 s2, s0, 16 ; GCNX3-HSA-NEXT: s_addc_u32 s3, s1, 0 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v9, s3 +; GCNX3-HSA-NEXT: v_mov_b32_e32 v7, v5 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v8, s2 ; GCNX3-HSA-NEXT: s_waitcnt vmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v4, v2 @@ -1800,6 +1851,9 @@ define amdgpu_kernel void @global_sextload_v4i32_to_v4i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_sextload_v4i32_to_v4i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1941,8 +1995,10 @@ define amdgpu_kernel void @global_zextload_v8i32_to_v8i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_zextload_v8i32_to_v8i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v9, 0 -; GCNX3-HSA-NEXT: v_mov_b32_e32 v11, v9 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -1957,6 +2013,7 @@ define amdgpu_kernel void @global_zextload_v8i32_to_v8i64(ptr addrspace(1) %out, ; GCNX3-HSA-NEXT: v_mov_b32_e32 v15, s3 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v14, s2 ; GCNX3-HSA-NEXT: s_add_u32 s2, s0, 48 +; GCNX3-HSA-NEXT: v_mov_b32_e32 v11, v9 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v13, s1 ; GCNX3-HSA-NEXT: s_addc_u32 s3, s1, 0 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v12, s0 @@ -2134,6 +2191,9 @@ define amdgpu_kernel void @global_sextload_v8i32_to_v8i64(ptr addrspace(1) %out, ; GCNX3-HSA-LABEL: global_sextload_v8i32_to_v8i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -2370,6 +2430,9 @@ define amdgpu_kernel void @global_sextload_v16i32_to_v16i64(ptr addrspace(1) %ou ; GCNX3-HSA-LABEL: global_sextload_v16i32_to_v16i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -2731,8 +2794,10 @@ define amdgpu_kernel void @global_zextload_v16i32_to_v16i64(ptr addrspace(1) %ou ; GCNX3-HSA-LABEL: global_zextload_v16i32_to_v16i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v17, 0 -; GCNX3-HSA-NEXT: v_mov_b32_e32 v19, v17 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCNX3-HSA-NEXT: s_addc_u32 s5, s3, 0 @@ -2766,6 +2831,7 @@ define amdgpu_kernel void @global_zextload_v16i32_to_v16i64(ptr addrspace(1) %ou ; GCNX3-HSA-NEXT: v_mov_b32_e32 v21, s1 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v26, s2 ; GCNX3-HSA-NEXT: s_add_u32 s2, s0, 64 +; GCNX3-HSA-NEXT: v_mov_b32_e32 v19, v17 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v20, s0 ; GCNX3-HSA-NEXT: s_addc_u32 s3, s1, 0 ; GCNX3-HSA-NEXT: s_waitcnt vmcnt(3) @@ -3122,6 +3188,9 @@ define amdgpu_kernel void @global_sextload_v32i32_to_v32i64(ptr addrspace(1) %ou ; GCNX3-HSA-LABEL: global_sextload_v32i32_to_v32i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: v_mov_b32_e32 v0, s2 ; GCNX3-HSA-NEXT: v_mov_b32_e32 v1, s3 @@ -3589,12 +3658,12 @@ define amdgpu_kernel void @global_sextload_v32i32_to_v32i64(ptr addrspace(1) %ou ; ; GCN-GFX900-HSA-LABEL: global_sextload_v32i32_to_v32i64: ; GCN-GFX900-HSA: ; %bb.0: -; GCN-GFX900-HSA-NEXT: s_mov_b64 s[18:19], s[2:3] -; GCN-GFX900-HSA-NEXT: s_mov_b64 s[16:17], s[0:1] +; GCN-GFX900-HSA-NEXT: s_mov_b64 s[22:23], s[2:3] +; GCN-GFX900-HSA-NEXT: s_mov_b64 s[20:21], s[0:1] ; GCN-GFX900-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GCN-GFX900-HSA-NEXT: v_mov_b32_e32 v8, 0 -; GCN-GFX900-HSA-NEXT: s_add_u32 s16, s16, s15 -; GCN-GFX900-HSA-NEXT: s_addc_u32 s17, s17, 0 +; GCN-GFX900-HSA-NEXT: s_add_u32 s20, s20, s17 +; GCN-GFX900-HSA-NEXT: s_addc_u32 s21, s21, 0 ; GCN-GFX900-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCN-GFX900-HSA-NEXT: global_load_dwordx4 v[0:3], v8, s[2:3] offset:96 ; GCN-GFX900-HSA-NEXT: global_load_dwordx4 v[4:7], v8, s[2:3] offset:112 @@ -3620,11 +3689,11 @@ define amdgpu_kernel void @global_sextload_v32i32_to_v32i64(ptr addrspace(1) %ou ; GCN-GFX900-HSA-NEXT: v_ashrrev_i32_e32 v5, 31, v0 ; GCN-GFX900-HSA-NEXT: v_mov_b32_e32 v4, v0 ; GCN-GFX900-HSA-NEXT: v_mov_b32_e32 v6, v1 -; GCN-GFX900-HSA-NEXT: buffer_store_dword v25, off, s[16:19], 0 ; 4-byte Folded Spill +; GCN-GFX900-HSA-NEXT: buffer_store_dword v25, off, s[20:23], 0 ; 4-byte Folded Spill ; GCN-GFX900-HSA-NEXT: s_nop 0 -; GCN-GFX900-HSA-NEXT: buffer_store_dword v26, off, s[16:19], 0 offset:4 ; 4-byte Folded Spill -; GCN-GFX900-HSA-NEXT: buffer_store_dword v27, off, s[16:19], 0 offset:8 ; 4-byte Folded Spill -; GCN-GFX900-HSA-NEXT: buffer_store_dword v28, off, s[16:19], 0 offset:12 ; 4-byte Folded Spill +; GCN-GFX900-HSA-NEXT: buffer_store_dword v26, off, s[20:23], 0 offset:4 ; 4-byte Folded Spill +; GCN-GFX900-HSA-NEXT: buffer_store_dword v27, off, s[20:23], 0 offset:8 ; 4-byte Folded Spill +; GCN-GFX900-HSA-NEXT: buffer_store_dword v28, off, s[20:23], 0 offset:12 ; 4-byte Folded Spill ; GCN-GFX900-HSA-NEXT: s_waitcnt vmcnt(7) ; GCN-GFX900-HSA-NEXT: v_ashrrev_i32_e32 v28, 31, v12 ; GCN-GFX900-HSA-NEXT: v_ashrrev_i32_e32 v26, 31, v11 @@ -3667,11 +3736,11 @@ define amdgpu_kernel void @global_sextload_v32i32_to_v32i64(ptr addrspace(1) %ou ; GCN-GFX900-HSA-NEXT: global_store_dwordx4 v8, v[33:36], s[0:1] offset:224 ; GCN-GFX900-HSA-NEXT: global_store_dwordx4 v8, v[29:32], s[0:1] offset:240 ; GCN-GFX900-HSA-NEXT: global_store_dwordx4 v8, v[4:7], s[0:1] offset:192 -; GCN-GFX900-HSA-NEXT: buffer_load_dword v32, off, s[16:19], 0 ; 4-byte Folded Reload +; GCN-GFX900-HSA-NEXT: buffer_load_dword v32, off, s[20:23], 0 ; 4-byte Folded Reload ; GCN-GFX900-HSA-NEXT: s_nop 0 -; GCN-GFX900-HSA-NEXT: buffer_load_dword v33, off, s[16:19], 0 offset:4 ; 4-byte Folded Reload -; GCN-GFX900-HSA-NEXT: buffer_load_dword v34, off, s[16:19], 0 offset:8 ; 4-byte Folded Reload -; GCN-GFX900-HSA-NEXT: buffer_load_dword v35, off, s[16:19], 0 offset:12 ; 4-byte Folded Reload +; GCN-GFX900-HSA-NEXT: buffer_load_dword v33, off, s[20:23], 0 offset:4 ; 4-byte Folded Reload +; GCN-GFX900-HSA-NEXT: buffer_load_dword v34, off, s[20:23], 0 offset:8 ; 4-byte Folded Reload +; GCN-GFX900-HSA-NEXT: buffer_load_dword v35, off, s[20:23], 0 offset:12 ; 4-byte Folded Reload ; GCN-GFX900-HSA-NEXT: s_waitcnt vmcnt(8) ; GCN-GFX900-HSA-NEXT: v_ashrrev_i32_e32 v60, 31, v52 ; GCN-GFX900-HSA-NEXT: v_ashrrev_i32_e32 v58, 31, v51 @@ -3913,6 +3982,9 @@ define amdgpu_kernel void @global_zextload_v32i32_to_v32i64(ptr addrspace(1) %ou ; GCNX3-HSA-LABEL: global_zextload_v32i32_to_v32i64: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCNX3-HSA-NEXT: s_addc_u32 s5, s3, 0 @@ -4437,6 +4509,9 @@ define amdgpu_kernel void @global_load_v32i32(ptr addrspace(1) %out, ptr addrspa ; GCNX3-HSA-LABEL: global_load_v32i32: ; GCNX3-HSA: ; %bb.0: ; GCNX3-HSA-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCNX3-HSA-NEXT: s_add_i32 s12, s12, s17 +; GCNX3-HSA-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCNX3-HSA-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCNX3-HSA-NEXT: s_waitcnt lgkmcnt(0) ; GCNX3-HSA-NEXT: s_add_u32 s4, s2, 16 ; GCNX3-HSA-NEXT: s_addc_u32 s5, s3, 0 diff --git a/llvm/test/CodeGen/AMDGPU/load-select-ptr.ll b/llvm/test/CodeGen/AMDGPU/load-select-ptr.ll index 4dfc773d615e4..1a6fa3c518ca7 100644 --- a/llvm/test/CodeGen/AMDGPU/load-select-ptr.ll +++ b/llvm/test/CodeGen/AMDGPU/load-select-ptr.ll @@ -13,7 +13,8 @@ ; GCN: s_cselect_b32 ; GCN-NOT: load_dword -; GCN: flat_load_dwordx2 +; GCN: flat_load_dword +; GCN: flat_load_dword ; GCN-NOT: load_dword ; GCN: flat_store_dwordx2 diff --git a/llvm/test/CodeGen/AMDGPU/mad24-get-global-id.ll b/llvm/test/CodeGen/AMDGPU/mad24-get-global-id.ll index 245a2775d9f2f..07b5e1610cfc0 100644 --- a/llvm/test/CodeGen/AMDGPU/mad24-get-global-id.ll +++ b/llvm/test/CodeGen/AMDGPU/mad24-get-global-id.ll @@ -9,7 +9,7 @@ declare ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() #0 ; GCN-LABEL: {{^}}get_global_id_0: ; GCN: s_and_b32 [[WGSIZEX:s[0-9]+]], {{s[0-9]+}}, 0xffff -; GCN: s_mul_i32 [[MUL:s[0-9]+]], s12, [[WGSIZEX]] +; GCN: s_mul_i32 [[MUL:s[0-9]+]], s14, [[WGSIZEX]] ; GCN: v_add_i32_e32 v{{[0-9]+}}, vcc, [[MUL]], v0 define amdgpu_kernel void @get_global_id_0(ptr addrspace(1) %out) #1 { %dispatch.ptr = call ptr addrspace(4) @llvm.amdgcn.dispatch.ptr() diff --git a/llvm/test/CodeGen/AMDGPU/match-perm-extract-vector-elt-bug.ll b/llvm/test/CodeGen/AMDGPU/match-perm-extract-vector-elt-bug.ll index b3b529d4e5e5b..4896e504cfdf4 100644 --- a/llvm/test/CodeGen/AMDGPU/match-perm-extract-vector-elt-bug.ll +++ b/llvm/test/CodeGen/AMDGPU/match-perm-extract-vector-elt-bug.ll @@ -11,8 +11,8 @@ define amdgpu_kernel void @test(ptr addrspace(1) %src, ptr addrspace(1) %dst) { ; GFX9-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX9-NEXT: s_waitcnt lgkmcnt(0) ; GFX9-NEXT: s_and_b32 s4, s4, 0xffff -; GFX9-NEXT: s_mul_i32 s12, s12, s4 -; GFX9-NEXT: s_add_i32 s5, s5, s12 +; GFX9-NEXT: s_mul_i32 s14, s14, s4 +; GFX9-NEXT: s_add_i32 s5, s5, s14 ; GFX9-NEXT: v_add_u32_e32 v0, s5, v0 ; GFX9-NEXT: v_ashrrev_i32_e32 v1, 31, v0 ; GFX9-NEXT: v_lshlrev_b64 v[4:5], 4, v[0:1] @@ -39,8 +39,8 @@ define amdgpu_kernel void @test(ptr addrspace(1) %src, ptr addrspace(1) %dst) { ; GFX10-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX10-NEXT: s_waitcnt lgkmcnt(0) ; GFX10-NEXT: s_and_b32 s4, s4, 0xffff -; GFX10-NEXT: s_mul_i32 s12, s12, s4 -; GFX10-NEXT: v_add3_u32 v0, s5, s12, v0 +; GFX10-NEXT: s_mul_i32 s14, s14, s4 +; GFX10-NEXT: v_add3_u32 v0, s5, s14, v0 ; GFX10-NEXT: v_ashrrev_i32_e32 v1, 31, v0 ; GFX10-NEXT: v_lshlrev_b64 v[4:5], 4, v[0:1] ; GFX10-NEXT: v_add_co_u32 v0, vcc_lo, s0, v4 diff --git a/llvm/test/CodeGen/AMDGPU/memcpy-libcall.ll b/llvm/test/CodeGen/AMDGPU/memcpy-libcall.ll index 88ee2a34dd49f..8d020b9e1a603 100644 --- a/llvm/test/CodeGen/AMDGPU/memcpy-libcall.ll +++ b/llvm/test/CodeGen/AMDGPU/memcpy-libcall.ll @@ -9,6 +9,8 @@ define amdgpu_kernel void @memcpy_p0_p0_minsize(ptr %dest, ptr readonly %src) #0 ; CHECK-LABEL: memcpy_p0_p0_minsize: ; CHECK: ; %bb.0: ; %entry ; CHECK-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v12, s3 ; CHECK-NEXT: v_mov_b32_e32 v11, s2 @@ -94,12 +96,12 @@ entry: define amdgpu_kernel void @memcpy_p5_p4_minsize(ptr addrspace(5) %local, ptr addrspace(4) %0) #0 { ; CHECK-LABEL: memcpy_p5_p4_minsize: ; CHECK: ; %bb.0: ; %entry -; CHECK-NEXT: s_mov_b64 s[18:19], s[2:3] -; CHECK-NEXT: s_mov_b64 s[16:17], s[0:1] +; CHECK-NEXT: s_mov_b64 s[22:23], s[2:3] +; CHECK-NEXT: s_mov_b64 s[20:21], s[0:1] ; CHECK-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x8 ; CHECK-NEXT: s_load_dword s2, s[8:9], 0x0 ; CHECK-NEXT: v_mov_b32_e32 v24, 0 -; CHECK-NEXT: s_add_u32 s16, s16, s15 +; CHECK-NEXT: s_add_u32 s20, s20, s17 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: global_load_dwordx4 v[0:3], v24, s[0:1] offset:112 ; CHECK-NEXT: global_load_dwordx4 v[4:7], v24, s[0:1] offset:96 @@ -107,50 +109,50 @@ define amdgpu_kernel void @memcpy_p5_p4_minsize(ptr addrspace(5) %local, ptr add ; CHECK-NEXT: global_load_dwordx4 v[12:15], v24, s[0:1] offset:64 ; CHECK-NEXT: global_load_dwordx4 v[16:19], v24, s[0:1] offset:48 ; CHECK-NEXT: global_load_dwordx4 v[20:23], v24, s[0:1] offset:32 -; CHECK-NEXT: s_addc_u32 s17, s17, 0 +; CHECK-NEXT: s_addc_u32 s21, s21, 0 ; CHECK-NEXT: v_mov_b32_e32 v25, s2 ; CHECK-NEXT: s_waitcnt vmcnt(5) -; CHECK-NEXT: buffer_store_dword v3, v25, s[16:19], 0 offen offset:124 -; CHECK-NEXT: buffer_store_dword v2, v25, s[16:19], 0 offen offset:120 -; CHECK-NEXT: buffer_store_dword v1, v25, s[16:19], 0 offen offset:116 -; CHECK-NEXT: buffer_store_dword v0, v25, s[16:19], 0 offen offset:112 +; CHECK-NEXT: buffer_store_dword v3, v25, s[20:23], 0 offen offset:124 +; CHECK-NEXT: buffer_store_dword v2, v25, s[20:23], 0 offen offset:120 +; CHECK-NEXT: buffer_store_dword v1, v25, s[20:23], 0 offen offset:116 +; CHECK-NEXT: buffer_store_dword v0, v25, s[20:23], 0 offen offset:112 ; CHECK-NEXT: global_load_dwordx4 v[0:3], v24, s[0:1] offset:16 ; CHECK-NEXT: s_waitcnt vmcnt(9) -; CHECK-NEXT: buffer_store_dword v7, v25, s[16:19], 0 offen offset:108 -; CHECK-NEXT: buffer_store_dword v6, v25, s[16:19], 0 offen offset:104 -; CHECK-NEXT: buffer_store_dword v5, v25, s[16:19], 0 offen offset:100 -; CHECK-NEXT: buffer_store_dword v4, v25, s[16:19], 0 offen offset:96 +; CHECK-NEXT: buffer_store_dword v7, v25, s[20:23], 0 offen offset:108 +; CHECK-NEXT: buffer_store_dword v6, v25, s[20:23], 0 offen offset:104 +; CHECK-NEXT: buffer_store_dword v5, v25, s[20:23], 0 offen offset:100 +; CHECK-NEXT: buffer_store_dword v4, v25, s[20:23], 0 offen offset:96 ; CHECK-NEXT: global_load_dwordx4 v[4:7], v24, s[0:1] ; CHECK-NEXT: s_waitcnt vmcnt(13) -; CHECK-NEXT: buffer_store_dword v11, v25, s[16:19], 0 offen offset:92 -; CHECK-NEXT: buffer_store_dword v10, v25, s[16:19], 0 offen offset:88 -; CHECK-NEXT: buffer_store_dword v9, v25, s[16:19], 0 offen offset:84 -; CHECK-NEXT: buffer_store_dword v8, v25, s[16:19], 0 offen offset:80 +; CHECK-NEXT: buffer_store_dword v11, v25, s[20:23], 0 offen offset:92 +; CHECK-NEXT: buffer_store_dword v10, v25, s[20:23], 0 offen offset:88 +; CHECK-NEXT: buffer_store_dword v9, v25, s[20:23], 0 offen offset:84 +; CHECK-NEXT: buffer_store_dword v8, v25, s[20:23], 0 offen offset:80 ; CHECK-NEXT: s_waitcnt vmcnt(16) -; CHECK-NEXT: buffer_store_dword v15, v25, s[16:19], 0 offen offset:76 -; CHECK-NEXT: buffer_store_dword v14, v25, s[16:19], 0 offen offset:72 -; CHECK-NEXT: buffer_store_dword v13, v25, s[16:19], 0 offen offset:68 -; CHECK-NEXT: buffer_store_dword v12, v25, s[16:19], 0 offen offset:64 +; CHECK-NEXT: buffer_store_dword v15, v25, s[20:23], 0 offen offset:76 +; CHECK-NEXT: buffer_store_dword v14, v25, s[20:23], 0 offen offset:72 +; CHECK-NEXT: buffer_store_dword v13, v25, s[20:23], 0 offen offset:68 +; CHECK-NEXT: buffer_store_dword v12, v25, s[20:23], 0 offen offset:64 ; CHECK-NEXT: s_waitcnt vmcnt(19) -; CHECK-NEXT: buffer_store_dword v19, v25, s[16:19], 0 offen offset:60 -; CHECK-NEXT: buffer_store_dword v18, v25, s[16:19], 0 offen offset:56 -; CHECK-NEXT: buffer_store_dword v17, v25, s[16:19], 0 offen offset:52 -; CHECK-NEXT: buffer_store_dword v16, v25, s[16:19], 0 offen offset:48 +; CHECK-NEXT: buffer_store_dword v19, v25, s[20:23], 0 offen offset:60 +; CHECK-NEXT: buffer_store_dword v18, v25, s[20:23], 0 offen offset:56 +; CHECK-NEXT: buffer_store_dword v17, v25, s[20:23], 0 offen offset:52 +; CHECK-NEXT: buffer_store_dword v16, v25, s[20:23], 0 offen offset:48 ; CHECK-NEXT: s_waitcnt vmcnt(22) -; CHECK-NEXT: buffer_store_dword v23, v25, s[16:19], 0 offen offset:44 -; CHECK-NEXT: buffer_store_dword v22, v25, s[16:19], 0 offen offset:40 -; CHECK-NEXT: buffer_store_dword v21, v25, s[16:19], 0 offen offset:36 -; CHECK-NEXT: buffer_store_dword v20, v25, s[16:19], 0 offen offset:32 +; CHECK-NEXT: buffer_store_dword v23, v25, s[20:23], 0 offen offset:44 +; CHECK-NEXT: buffer_store_dword v22, v25, s[20:23], 0 offen offset:40 +; CHECK-NEXT: buffer_store_dword v21, v25, s[20:23], 0 offen offset:36 +; CHECK-NEXT: buffer_store_dword v20, v25, s[20:23], 0 offen offset:32 ; CHECK-NEXT: s_waitcnt vmcnt(21) -; CHECK-NEXT: buffer_store_dword v3, v25, s[16:19], 0 offen offset:28 -; CHECK-NEXT: buffer_store_dword v2, v25, s[16:19], 0 offen offset:24 -; CHECK-NEXT: buffer_store_dword v1, v25, s[16:19], 0 offen offset:20 -; CHECK-NEXT: buffer_store_dword v0, v25, s[16:19], 0 offen offset:16 +; CHECK-NEXT: buffer_store_dword v3, v25, s[20:23], 0 offen offset:28 +; CHECK-NEXT: buffer_store_dword v2, v25, s[20:23], 0 offen offset:24 +; CHECK-NEXT: buffer_store_dword v1, v25, s[20:23], 0 offen offset:20 +; CHECK-NEXT: buffer_store_dword v0, v25, s[20:23], 0 offen offset:16 ; CHECK-NEXT: s_waitcnt vmcnt(20) -; CHECK-NEXT: buffer_store_dword v7, v25, s[16:19], 0 offen offset:12 -; CHECK-NEXT: buffer_store_dword v6, v25, s[16:19], 0 offen offset:8 -; CHECK-NEXT: buffer_store_dword v5, v25, s[16:19], 0 offen offset:4 -; CHECK-NEXT: buffer_store_dword v4, v25, s[16:19], 0 offen +; CHECK-NEXT: buffer_store_dword v7, v25, s[20:23], 0 offen offset:12 +; CHECK-NEXT: buffer_store_dword v6, v25, s[20:23], 0 offen offset:8 +; CHECK-NEXT: buffer_store_dword v5, v25, s[20:23], 0 offen offset:4 +; CHECK-NEXT: buffer_store_dword v4, v25, s[20:23], 0 offen ; CHECK-NEXT: s_endpgm entry: tail call void @llvm.memcpy.p5.p4.i64(ptr addrspace(5) %local, ptr addrspace(4) %0, i64 128, i1 false) @@ -160,55 +162,57 @@ entry: define amdgpu_kernel void @memcpy_p0_p5_minsize(ptr %generic, ptr addrspace(5) %src) #0 { ; CHECK-LABEL: memcpy_p0_p5_minsize: ; CHECK: ; %bb.0: ; %entry -; CHECK-NEXT: s_mov_b64 s[18:19], s[2:3] -; CHECK-NEXT: s_mov_b64 s[16:17], s[0:1] +; CHECK-NEXT: s_mov_b64 s[22:23], s[2:3] +; CHECK-NEXT: s_mov_b64 s[20:21], s[0:1] ; CHECK-NEXT: s_load_dword s0, s[8:9], 0x8 -; CHECK-NEXT: s_add_u32 s16, s16, s15 -; CHECK-NEXT: s_addc_u32 s17, s17, 0 +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 +; CHECK-NEXT: s_add_u32 s20, s20, s17 +; CHECK-NEXT: s_addc_u32 s21, s21, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v26, s0 -; CHECK-NEXT: buffer_load_dword v3, v26, s[16:19], 0 offen offset:124 -; CHECK-NEXT: buffer_load_dword v2, v26, s[16:19], 0 offen offset:120 -; CHECK-NEXT: buffer_load_dword v1, v26, s[16:19], 0 offen offset:116 -; CHECK-NEXT: buffer_load_dword v0, v26, s[16:19], 0 offen offset:112 -; CHECK-NEXT: buffer_load_dword v7, v26, s[16:19], 0 offen offset:108 -; CHECK-NEXT: buffer_load_dword v6, v26, s[16:19], 0 offen offset:104 -; CHECK-NEXT: buffer_load_dword v5, v26, s[16:19], 0 offen offset:100 -; CHECK-NEXT: buffer_load_dword v4, v26, s[16:19], 0 offen offset:96 +; CHECK-NEXT: buffer_load_dword v3, v26, s[20:23], 0 offen offset:124 +; CHECK-NEXT: buffer_load_dword v2, v26, s[20:23], 0 offen offset:120 +; CHECK-NEXT: buffer_load_dword v1, v26, s[20:23], 0 offen offset:116 +; CHECK-NEXT: buffer_load_dword v0, v26, s[20:23], 0 offen offset:112 +; CHECK-NEXT: buffer_load_dword v7, v26, s[20:23], 0 offen offset:108 +; CHECK-NEXT: buffer_load_dword v6, v26, s[20:23], 0 offen offset:104 +; CHECK-NEXT: buffer_load_dword v5, v26, s[20:23], 0 offen offset:100 +; CHECK-NEXT: buffer_load_dword v4, v26, s[20:23], 0 offen offset:96 ; CHECK-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; CHECK-NEXT: buffer_load_dword v8, v26, s[16:19], 0 offen offset:16 -; CHECK-NEXT: buffer_load_dword v9, v26, s[16:19], 0 offen offset:20 -; CHECK-NEXT: buffer_load_dword v10, v26, s[16:19], 0 offen offset:24 -; CHECK-NEXT: buffer_load_dword v11, v26, s[16:19], 0 offen offset:28 -; CHECK-NEXT: buffer_load_dword v12, v26, s[16:19], 0 offen offset:32 -; CHECK-NEXT: buffer_load_dword v13, v26, s[16:19], 0 offen offset:36 -; CHECK-NEXT: buffer_load_dword v14, v26, s[16:19], 0 offen offset:40 -; CHECK-NEXT: buffer_load_dword v15, v26, s[16:19], 0 offen offset:44 -; CHECK-NEXT: buffer_load_dword v16, v26, s[16:19], 0 offen offset:48 -; CHECK-NEXT: buffer_load_dword v17, v26, s[16:19], 0 offen offset:52 -; CHECK-NEXT: buffer_load_dword v18, v26, s[16:19], 0 offen offset:56 -; CHECK-NEXT: buffer_load_dword v19, v26, s[16:19], 0 offen offset:60 -; CHECK-NEXT: buffer_load_dword v23, v26, s[16:19], 0 offen offset:92 -; CHECK-NEXT: buffer_load_dword v22, v26, s[16:19], 0 offen offset:88 -; CHECK-NEXT: buffer_load_dword v21, v26, s[16:19], 0 offen offset:84 -; CHECK-NEXT: buffer_load_dword v20, v26, s[16:19], 0 offen offset:80 +; CHECK-NEXT: buffer_load_dword v8, v26, s[20:23], 0 offen offset:16 +; CHECK-NEXT: buffer_load_dword v9, v26, s[20:23], 0 offen offset:20 +; CHECK-NEXT: buffer_load_dword v10, v26, s[20:23], 0 offen offset:24 +; CHECK-NEXT: buffer_load_dword v11, v26, s[20:23], 0 offen offset:28 +; CHECK-NEXT: buffer_load_dword v12, v26, s[20:23], 0 offen offset:32 +; CHECK-NEXT: buffer_load_dword v13, v26, s[20:23], 0 offen offset:36 +; CHECK-NEXT: buffer_load_dword v14, v26, s[20:23], 0 offen offset:40 +; CHECK-NEXT: buffer_load_dword v15, v26, s[20:23], 0 offen offset:44 +; CHECK-NEXT: buffer_load_dword v16, v26, s[20:23], 0 offen offset:48 +; CHECK-NEXT: buffer_load_dword v17, v26, s[20:23], 0 offen offset:52 +; CHECK-NEXT: buffer_load_dword v18, v26, s[20:23], 0 offen offset:56 +; CHECK-NEXT: buffer_load_dword v19, v26, s[20:23], 0 offen offset:60 +; CHECK-NEXT: buffer_load_dword v23, v26, s[20:23], 0 offen offset:92 +; CHECK-NEXT: buffer_load_dword v22, v26, s[20:23], 0 offen offset:88 +; CHECK-NEXT: buffer_load_dword v21, v26, s[20:23], 0 offen offset:84 +; CHECK-NEXT: buffer_load_dword v20, v26, s[20:23], 0 offen offset:80 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v25, s1 ; CHECK-NEXT: v_mov_b32_e32 v24, s0 ; CHECK-NEXT: s_waitcnt vmcnt(20) ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[0:3] offset:112 -; CHECK-NEXT: buffer_load_dword v3, v26, s[16:19], 0 offen offset:76 +; CHECK-NEXT: buffer_load_dword v3, v26, s[20:23], 0 offen offset:76 ; CHECK-NEXT: s_nop 0 -; CHECK-NEXT: buffer_load_dword v2, v26, s[16:19], 0 offen offset:72 -; CHECK-NEXT: buffer_load_dword v1, v26, s[16:19], 0 offen offset:68 -; CHECK-NEXT: buffer_load_dword v0, v26, s[16:19], 0 offen offset:64 +; CHECK-NEXT: buffer_load_dword v2, v26, s[20:23], 0 offen offset:72 +; CHECK-NEXT: buffer_load_dword v1, v26, s[20:23], 0 offen offset:68 +; CHECK-NEXT: buffer_load_dword v0, v26, s[20:23], 0 offen offset:64 ; CHECK-NEXT: s_waitcnt vmcnt(0) ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[4:7] offset:96 -; CHECK-NEXT: buffer_load_dword v4, v26, s[16:19], 0 offen +; CHECK-NEXT: buffer_load_dword v4, v26, s[20:23], 0 offen ; CHECK-NEXT: s_nop 0 -; CHECK-NEXT: buffer_load_dword v5, v26, s[16:19], 0 offen offset:4 -; CHECK-NEXT: buffer_load_dword v6, v26, s[16:19], 0 offen offset:8 -; CHECK-NEXT: buffer_load_dword v7, v26, s[16:19], 0 offen offset:12 +; CHECK-NEXT: buffer_load_dword v5, v26, s[20:23], 0 offen offset:4 +; CHECK-NEXT: buffer_load_dword v6, v26, s[20:23], 0 offen offset:8 +; CHECK-NEXT: buffer_load_dword v7, v26, s[20:23], 0 offen offset:12 ; CHECK-NEXT: s_nop 0 ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[20:23] offset:80 ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[0:3] offset:64 @@ -268,6 +272,8 @@ define amdgpu_kernel void @memcpy_p0_p3_minsize(ptr %generic) #0 { ; CHECK-NEXT: ds_read2_b64 v[4:7], v16 offset0:2 offset1:3 ; CHECK-NEXT: ds_read2_b64 v[8:11], v16 offset0:4 offset1:5 ; CHECK-NEXT: ds_read2_b64 v[12:15], v16 offset0:6 offset1:7 +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v21, s1 ; CHECK-NEXT: v_mov_b32_e32 v20, s0 @@ -294,6 +300,8 @@ define amdgpu_kernel void @memcpy_p0_p0_optsize(ptr %dest, ptr %src) #1 { ; CHECK-LABEL: memcpy_p0_p0_optsize: ; CHECK: ; %bb.0: ; %entry ; CHECK-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v12, s3 ; CHECK-NEXT: v_mov_b32_e32 v11, s2 @@ -379,12 +387,12 @@ entry: define amdgpu_kernel void @memcpy_p5_p4_optsize(ptr addrspace(5) %local, ptr addrspace(4) %0) #1 { ; CHECK-LABEL: memcpy_p5_p4_optsize: ; CHECK: ; %bb.0: ; %entry -; CHECK-NEXT: s_mov_b64 s[18:19], s[2:3] -; CHECK-NEXT: s_mov_b64 s[16:17], s[0:1] +; CHECK-NEXT: s_mov_b64 s[22:23], s[2:3] +; CHECK-NEXT: s_mov_b64 s[20:21], s[0:1] ; CHECK-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x8 ; CHECK-NEXT: s_load_dword s2, s[8:9], 0x0 ; CHECK-NEXT: v_mov_b32_e32 v24, 0 -; CHECK-NEXT: s_add_u32 s16, s16, s15 +; CHECK-NEXT: s_add_u32 s20, s20, s17 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: global_load_dwordx4 v[0:3], v24, s[0:1] offset:112 ; CHECK-NEXT: global_load_dwordx4 v[4:7], v24, s[0:1] offset:96 @@ -392,50 +400,50 @@ define amdgpu_kernel void @memcpy_p5_p4_optsize(ptr addrspace(5) %local, ptr add ; CHECK-NEXT: global_load_dwordx4 v[12:15], v24, s[0:1] offset:64 ; CHECK-NEXT: global_load_dwordx4 v[16:19], v24, s[0:1] offset:48 ; CHECK-NEXT: global_load_dwordx4 v[20:23], v24, s[0:1] offset:32 -; CHECK-NEXT: s_addc_u32 s17, s17, 0 +; CHECK-NEXT: s_addc_u32 s21, s21, 0 ; CHECK-NEXT: v_mov_b32_e32 v25, s2 ; CHECK-NEXT: s_waitcnt vmcnt(5) -; CHECK-NEXT: buffer_store_dword v3, v25, s[16:19], 0 offen offset:124 -; CHECK-NEXT: buffer_store_dword v2, v25, s[16:19], 0 offen offset:120 -; CHECK-NEXT: buffer_store_dword v1, v25, s[16:19], 0 offen offset:116 -; CHECK-NEXT: buffer_store_dword v0, v25, s[16:19], 0 offen offset:112 +; CHECK-NEXT: buffer_store_dword v3, v25, s[20:23], 0 offen offset:124 +; CHECK-NEXT: buffer_store_dword v2, v25, s[20:23], 0 offen offset:120 +; CHECK-NEXT: buffer_store_dword v1, v25, s[20:23], 0 offen offset:116 +; CHECK-NEXT: buffer_store_dword v0, v25, s[20:23], 0 offen offset:112 ; CHECK-NEXT: global_load_dwordx4 v[0:3], v24, s[0:1] offset:16 ; CHECK-NEXT: s_waitcnt vmcnt(9) -; CHECK-NEXT: buffer_store_dword v7, v25, s[16:19], 0 offen offset:108 -; CHECK-NEXT: buffer_store_dword v6, v25, s[16:19], 0 offen offset:104 -; CHECK-NEXT: buffer_store_dword v5, v25, s[16:19], 0 offen offset:100 -; CHECK-NEXT: buffer_store_dword v4, v25, s[16:19], 0 offen offset:96 +; CHECK-NEXT: buffer_store_dword v7, v25, s[20:23], 0 offen offset:108 +; CHECK-NEXT: buffer_store_dword v6, v25, s[20:23], 0 offen offset:104 +; CHECK-NEXT: buffer_store_dword v5, v25, s[20:23], 0 offen offset:100 +; CHECK-NEXT: buffer_store_dword v4, v25, s[20:23], 0 offen offset:96 ; CHECK-NEXT: global_load_dwordx4 v[4:7], v24, s[0:1] ; CHECK-NEXT: s_waitcnt vmcnt(13) -; CHECK-NEXT: buffer_store_dword v11, v25, s[16:19], 0 offen offset:92 -; CHECK-NEXT: buffer_store_dword v10, v25, s[16:19], 0 offen offset:88 -; CHECK-NEXT: buffer_store_dword v9, v25, s[16:19], 0 offen offset:84 -; CHECK-NEXT: buffer_store_dword v8, v25, s[16:19], 0 offen offset:80 +; CHECK-NEXT: buffer_store_dword v11, v25, s[20:23], 0 offen offset:92 +; CHECK-NEXT: buffer_store_dword v10, v25, s[20:23], 0 offen offset:88 +; CHECK-NEXT: buffer_store_dword v9, v25, s[20:23], 0 offen offset:84 +; CHECK-NEXT: buffer_store_dword v8, v25, s[20:23], 0 offen offset:80 ; CHECK-NEXT: s_waitcnt vmcnt(16) -; CHECK-NEXT: buffer_store_dword v15, v25, s[16:19], 0 offen offset:76 -; CHECK-NEXT: buffer_store_dword v14, v25, s[16:19], 0 offen offset:72 -; CHECK-NEXT: buffer_store_dword v13, v25, s[16:19], 0 offen offset:68 -; CHECK-NEXT: buffer_store_dword v12, v25, s[16:19], 0 offen offset:64 +; CHECK-NEXT: buffer_store_dword v15, v25, s[20:23], 0 offen offset:76 +; CHECK-NEXT: buffer_store_dword v14, v25, s[20:23], 0 offen offset:72 +; CHECK-NEXT: buffer_store_dword v13, v25, s[20:23], 0 offen offset:68 +; CHECK-NEXT: buffer_store_dword v12, v25, s[20:23], 0 offen offset:64 ; CHECK-NEXT: s_waitcnt vmcnt(19) -; CHECK-NEXT: buffer_store_dword v19, v25, s[16:19], 0 offen offset:60 -; CHECK-NEXT: buffer_store_dword v18, v25, s[16:19], 0 offen offset:56 -; CHECK-NEXT: buffer_store_dword v17, v25, s[16:19], 0 offen offset:52 -; CHECK-NEXT: buffer_store_dword v16, v25, s[16:19], 0 offen offset:48 +; CHECK-NEXT: buffer_store_dword v19, v25, s[20:23], 0 offen offset:60 +; CHECK-NEXT: buffer_store_dword v18, v25, s[20:23], 0 offen offset:56 +; CHECK-NEXT: buffer_store_dword v17, v25, s[20:23], 0 offen offset:52 +; CHECK-NEXT: buffer_store_dword v16, v25, s[20:23], 0 offen offset:48 ; CHECK-NEXT: s_waitcnt vmcnt(22) -; CHECK-NEXT: buffer_store_dword v23, v25, s[16:19], 0 offen offset:44 -; CHECK-NEXT: buffer_store_dword v22, v25, s[16:19], 0 offen offset:40 -; CHECK-NEXT: buffer_store_dword v21, v25, s[16:19], 0 offen offset:36 -; CHECK-NEXT: buffer_store_dword v20, v25, s[16:19], 0 offen offset:32 +; CHECK-NEXT: buffer_store_dword v23, v25, s[20:23], 0 offen offset:44 +; CHECK-NEXT: buffer_store_dword v22, v25, s[20:23], 0 offen offset:40 +; CHECK-NEXT: buffer_store_dword v21, v25, s[20:23], 0 offen offset:36 +; CHECK-NEXT: buffer_store_dword v20, v25, s[20:23], 0 offen offset:32 ; CHECK-NEXT: s_waitcnt vmcnt(21) -; CHECK-NEXT: buffer_store_dword v3, v25, s[16:19], 0 offen offset:28 -; CHECK-NEXT: buffer_store_dword v2, v25, s[16:19], 0 offen offset:24 -; CHECK-NEXT: buffer_store_dword v1, v25, s[16:19], 0 offen offset:20 -; CHECK-NEXT: buffer_store_dword v0, v25, s[16:19], 0 offen offset:16 +; CHECK-NEXT: buffer_store_dword v3, v25, s[20:23], 0 offen offset:28 +; CHECK-NEXT: buffer_store_dword v2, v25, s[20:23], 0 offen offset:24 +; CHECK-NEXT: buffer_store_dword v1, v25, s[20:23], 0 offen offset:20 +; CHECK-NEXT: buffer_store_dword v0, v25, s[20:23], 0 offen offset:16 ; CHECK-NEXT: s_waitcnt vmcnt(20) -; CHECK-NEXT: buffer_store_dword v7, v25, s[16:19], 0 offen offset:12 -; CHECK-NEXT: buffer_store_dword v6, v25, s[16:19], 0 offen offset:8 -; CHECK-NEXT: buffer_store_dword v5, v25, s[16:19], 0 offen offset:4 -; CHECK-NEXT: buffer_store_dword v4, v25, s[16:19], 0 offen +; CHECK-NEXT: buffer_store_dword v7, v25, s[20:23], 0 offen offset:12 +; CHECK-NEXT: buffer_store_dword v6, v25, s[20:23], 0 offen offset:8 +; CHECK-NEXT: buffer_store_dword v5, v25, s[20:23], 0 offen offset:4 +; CHECK-NEXT: buffer_store_dword v4, v25, s[20:23], 0 offen ; CHECK-NEXT: s_endpgm entry: tail call void @llvm.memcpy.p5.p4.i64(ptr addrspace(5) %local, ptr addrspace(4) %0, i64 128, i1 false) @@ -445,55 +453,57 @@ entry: define amdgpu_kernel void @memcpy_p0_p5_optsize(ptr %generic, ptr addrspace(5) %src) #1 { ; CHECK-LABEL: memcpy_p0_p5_optsize: ; CHECK: ; %bb.0: ; %entry -; CHECK-NEXT: s_mov_b64 s[18:19], s[2:3] -; CHECK-NEXT: s_mov_b64 s[16:17], s[0:1] +; CHECK-NEXT: s_mov_b64 s[22:23], s[2:3] +; CHECK-NEXT: s_mov_b64 s[20:21], s[0:1] ; CHECK-NEXT: s_load_dword s0, s[8:9], 0x8 -; CHECK-NEXT: s_add_u32 s16, s16, s15 -; CHECK-NEXT: s_addc_u32 s17, s17, 0 +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 +; CHECK-NEXT: s_add_u32 s20, s20, s17 +; CHECK-NEXT: s_addc_u32 s21, s21, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v26, s0 -; CHECK-NEXT: buffer_load_dword v3, v26, s[16:19], 0 offen offset:124 -; CHECK-NEXT: buffer_load_dword v2, v26, s[16:19], 0 offen offset:120 -; CHECK-NEXT: buffer_load_dword v1, v26, s[16:19], 0 offen offset:116 -; CHECK-NEXT: buffer_load_dword v0, v26, s[16:19], 0 offen offset:112 -; CHECK-NEXT: buffer_load_dword v7, v26, s[16:19], 0 offen offset:108 -; CHECK-NEXT: buffer_load_dword v6, v26, s[16:19], 0 offen offset:104 -; CHECK-NEXT: buffer_load_dword v5, v26, s[16:19], 0 offen offset:100 -; CHECK-NEXT: buffer_load_dword v4, v26, s[16:19], 0 offen offset:96 +; CHECK-NEXT: buffer_load_dword v3, v26, s[20:23], 0 offen offset:124 +; CHECK-NEXT: buffer_load_dword v2, v26, s[20:23], 0 offen offset:120 +; CHECK-NEXT: buffer_load_dword v1, v26, s[20:23], 0 offen offset:116 +; CHECK-NEXT: buffer_load_dword v0, v26, s[20:23], 0 offen offset:112 +; CHECK-NEXT: buffer_load_dword v7, v26, s[20:23], 0 offen offset:108 +; CHECK-NEXT: buffer_load_dword v6, v26, s[20:23], 0 offen offset:104 +; CHECK-NEXT: buffer_load_dword v5, v26, s[20:23], 0 offen offset:100 +; CHECK-NEXT: buffer_load_dword v4, v26, s[20:23], 0 offen offset:96 ; CHECK-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; CHECK-NEXT: buffer_load_dword v8, v26, s[16:19], 0 offen offset:16 -; CHECK-NEXT: buffer_load_dword v9, v26, s[16:19], 0 offen offset:20 -; CHECK-NEXT: buffer_load_dword v10, v26, s[16:19], 0 offen offset:24 -; CHECK-NEXT: buffer_load_dword v11, v26, s[16:19], 0 offen offset:28 -; CHECK-NEXT: buffer_load_dword v12, v26, s[16:19], 0 offen offset:32 -; CHECK-NEXT: buffer_load_dword v13, v26, s[16:19], 0 offen offset:36 -; CHECK-NEXT: buffer_load_dword v14, v26, s[16:19], 0 offen offset:40 -; CHECK-NEXT: buffer_load_dword v15, v26, s[16:19], 0 offen offset:44 -; CHECK-NEXT: buffer_load_dword v16, v26, s[16:19], 0 offen offset:48 -; CHECK-NEXT: buffer_load_dword v17, v26, s[16:19], 0 offen offset:52 -; CHECK-NEXT: buffer_load_dword v18, v26, s[16:19], 0 offen offset:56 -; CHECK-NEXT: buffer_load_dword v19, v26, s[16:19], 0 offen offset:60 -; CHECK-NEXT: buffer_load_dword v23, v26, s[16:19], 0 offen offset:92 -; CHECK-NEXT: buffer_load_dword v22, v26, s[16:19], 0 offen offset:88 -; CHECK-NEXT: buffer_load_dword v21, v26, s[16:19], 0 offen offset:84 -; CHECK-NEXT: buffer_load_dword v20, v26, s[16:19], 0 offen offset:80 +; CHECK-NEXT: buffer_load_dword v8, v26, s[20:23], 0 offen offset:16 +; CHECK-NEXT: buffer_load_dword v9, v26, s[20:23], 0 offen offset:20 +; CHECK-NEXT: buffer_load_dword v10, v26, s[20:23], 0 offen offset:24 +; CHECK-NEXT: buffer_load_dword v11, v26, s[20:23], 0 offen offset:28 +; CHECK-NEXT: buffer_load_dword v12, v26, s[20:23], 0 offen offset:32 +; CHECK-NEXT: buffer_load_dword v13, v26, s[20:23], 0 offen offset:36 +; CHECK-NEXT: buffer_load_dword v14, v26, s[20:23], 0 offen offset:40 +; CHECK-NEXT: buffer_load_dword v15, v26, s[20:23], 0 offen offset:44 +; CHECK-NEXT: buffer_load_dword v16, v26, s[20:23], 0 offen offset:48 +; CHECK-NEXT: buffer_load_dword v17, v26, s[20:23], 0 offen offset:52 +; CHECK-NEXT: buffer_load_dword v18, v26, s[20:23], 0 offen offset:56 +; CHECK-NEXT: buffer_load_dword v19, v26, s[20:23], 0 offen offset:60 +; CHECK-NEXT: buffer_load_dword v23, v26, s[20:23], 0 offen offset:92 +; CHECK-NEXT: buffer_load_dword v22, v26, s[20:23], 0 offen offset:88 +; CHECK-NEXT: buffer_load_dword v21, v26, s[20:23], 0 offen offset:84 +; CHECK-NEXT: buffer_load_dword v20, v26, s[20:23], 0 offen offset:80 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v25, s1 ; CHECK-NEXT: v_mov_b32_e32 v24, s0 ; CHECK-NEXT: s_waitcnt vmcnt(20) ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[0:3] offset:112 -; CHECK-NEXT: buffer_load_dword v3, v26, s[16:19], 0 offen offset:76 +; CHECK-NEXT: buffer_load_dword v3, v26, s[20:23], 0 offen offset:76 ; CHECK-NEXT: s_nop 0 -; CHECK-NEXT: buffer_load_dword v2, v26, s[16:19], 0 offen offset:72 -; CHECK-NEXT: buffer_load_dword v1, v26, s[16:19], 0 offen offset:68 -; CHECK-NEXT: buffer_load_dword v0, v26, s[16:19], 0 offen offset:64 +; CHECK-NEXT: buffer_load_dword v2, v26, s[20:23], 0 offen offset:72 +; CHECK-NEXT: buffer_load_dword v1, v26, s[20:23], 0 offen offset:68 +; CHECK-NEXT: buffer_load_dword v0, v26, s[20:23], 0 offen offset:64 ; CHECK-NEXT: s_waitcnt vmcnt(0) ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[4:7] offset:96 -; CHECK-NEXT: buffer_load_dword v4, v26, s[16:19], 0 offen +; CHECK-NEXT: buffer_load_dword v4, v26, s[20:23], 0 offen ; CHECK-NEXT: s_nop 0 -; CHECK-NEXT: buffer_load_dword v5, v26, s[16:19], 0 offen offset:4 -; CHECK-NEXT: buffer_load_dword v6, v26, s[16:19], 0 offen offset:8 -; CHECK-NEXT: buffer_load_dword v7, v26, s[16:19], 0 offen offset:12 +; CHECK-NEXT: buffer_load_dword v5, v26, s[20:23], 0 offen offset:4 +; CHECK-NEXT: buffer_load_dword v6, v26, s[20:23], 0 offen offset:8 +; CHECK-NEXT: buffer_load_dword v7, v26, s[20:23], 0 offen offset:12 ; CHECK-NEXT: s_nop 0 ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[20:23] offset:80 ; CHECK-NEXT: flat_store_dwordx4 v[24:25], v[0:3] offset:64 @@ -553,6 +563,8 @@ define amdgpu_kernel void @memcpy_p0_p3_optsize(ptr %generic) #1 { ; CHECK-NEXT: ds_read2_b64 v[4:7], v16 offset0:2 offset1:3 ; CHECK-NEXT: ds_read2_b64 v[8:11], v16 offset0:4 offset1:5 ; CHECK-NEXT: ds_read2_b64 v[12:15], v16 offset0:6 offset1:7 +; CHECK-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; CHECK-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: v_mov_b32_e32 v21, s1 ; CHECK-NEXT: v_mov_b32_e32 v20, s0 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-agent.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-agent.ll index 5af37809443e0..07ad8cb0c4a3d 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-agent.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-agent.ll @@ -15,6 +15,9 @@ define amdgpu_kernel void @flat_agent_unordered_load( ; GFX7-LABEL: flat_agent_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -29,6 +32,10 @@ define amdgpu_kernel void @flat_agent_unordered_load( ; ; GFX10-WGP-LABEL: flat_agent_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -43,6 +50,10 @@ define amdgpu_kernel void @flat_agent_unordered_load( ; ; GFX10-CU-LABEL: flat_agent_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -71,6 +82,8 @@ define amdgpu_kernel void @flat_agent_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -83,6 +96,8 @@ define amdgpu_kernel void @flat_agent_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -182,6 +197,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_load( ; GFX7-LABEL: flat_agent_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -196,6 +214,10 @@ define amdgpu_kernel void @flat_agent_monotonic_load( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -210,6 +232,10 @@ define amdgpu_kernel void @flat_agent_monotonic_load( ; ; GFX10-CU-LABEL: flat_agent_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -238,6 +264,8 @@ define amdgpu_kernel void @flat_agent_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -250,6 +278,8 @@ define amdgpu_kernel void @flat_agent_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -349,6 +379,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_load( ; GFX7-LABEL: flat_agent_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -364,6 +397,10 @@ define amdgpu_kernel void @flat_agent_acquire_load( ; ; GFX10-WGP-LABEL: flat_agent_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -380,6 +417,10 @@ define amdgpu_kernel void @flat_agent_acquire_load( ; ; GFX10-CU-LABEL: flat_agent_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -410,6 +451,8 @@ define amdgpu_kernel void @flat_agent_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -423,6 +466,8 @@ define amdgpu_kernel void @flat_agent_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -531,6 +576,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_load( ; GFX7-LABEL: flat_agent_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -547,6 +595,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -565,6 +617,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_load( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -598,6 +654,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -612,6 +670,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -739,6 +799,9 @@ entry: define amdgpu_kernel void @flat_agent_unordered_store( ; GFX7-LABEL: flat_agent_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -750,6 +813,10 @@ define amdgpu_kernel void @flat_agent_unordered_store( ; ; GFX10-WGP-LABEL: flat_agent_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -761,6 +828,10 @@ define amdgpu_kernel void @flat_agent_unordered_store( ; ; GFX10-CU-LABEL: flat_agent_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -783,6 +854,8 @@ define amdgpu_kernel void @flat_agent_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -793,6 +866,8 @@ define amdgpu_kernel void @flat_agent_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -873,6 +948,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_store( ; GFX7-LABEL: flat_agent_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -884,6 +962,10 @@ define amdgpu_kernel void @flat_agent_monotonic_store( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -895,6 +977,10 @@ define amdgpu_kernel void @flat_agent_monotonic_store( ; ; GFX10-CU-LABEL: flat_agent_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -917,6 +1003,8 @@ define amdgpu_kernel void @flat_agent_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -927,6 +1015,8 @@ define amdgpu_kernel void @flat_agent_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1007,6 +1097,9 @@ entry: define amdgpu_kernel void @flat_agent_release_store( ; GFX7-LABEL: flat_agent_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1019,6 +1112,10 @@ define amdgpu_kernel void @flat_agent_release_store( ; ; GFX10-WGP-LABEL: flat_agent_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1032,6 +1129,10 @@ define amdgpu_kernel void @flat_agent_release_store( ; ; GFX10-CU-LABEL: flat_agent_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1057,6 +1158,8 @@ define amdgpu_kernel void @flat_agent_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1068,6 +1171,8 @@ define amdgpu_kernel void @flat_agent_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1165,6 +1270,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_store( ; GFX7-LABEL: flat_agent_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1177,6 +1285,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1190,6 +1302,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_store( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1215,6 +1331,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1226,6 +1344,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1323,6 +1443,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_atomicrmw( ; GFX7-LABEL: flat_agent_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1334,6 +1457,10 @@ define amdgpu_kernel void @flat_agent_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1345,6 +1472,10 @@ define amdgpu_kernel void @flat_agent_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1367,6 +1498,8 @@ define amdgpu_kernel void @flat_agent_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1377,6 +1510,8 @@ define amdgpu_kernel void @flat_agent_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1457,6 +1592,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_atomicrmw( ; GFX7-LABEL: flat_agent_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1470,6 +1608,10 @@ define amdgpu_kernel void @flat_agent_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1485,6 +1627,10 @@ define amdgpu_kernel void @flat_agent_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1512,6 +1658,8 @@ define amdgpu_kernel void @flat_agent_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1524,6 +1672,8 @@ define amdgpu_kernel void @flat_agent_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1622,6 +1772,9 @@ entry: define amdgpu_kernel void @flat_agent_release_atomicrmw( ; GFX7-LABEL: flat_agent_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1634,6 +1787,10 @@ define amdgpu_kernel void @flat_agent_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1647,6 +1804,10 @@ define amdgpu_kernel void @flat_agent_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1672,6 +1833,8 @@ define amdgpu_kernel void @flat_agent_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1683,6 +1846,8 @@ define amdgpu_kernel void @flat_agent_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1780,6 +1945,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_atomicrmw( ; GFX7-LABEL: flat_agent_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1794,6 +1962,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1811,6 +1983,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1841,6 +2017,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1854,6 +2032,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1969,6 +2149,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_atomicrmw( ; GFX7-LABEL: flat_agent_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1983,6 +2166,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2000,6 +2187,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2030,6 +2221,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2043,6 +2236,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2158,6 +2353,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_agent_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2174,6 +2372,10 @@ define amdgpu_kernel void @flat_agent_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2191,6 +2393,10 @@ define amdgpu_kernel void @flat_agent_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2223,6 +2429,8 @@ define amdgpu_kernel void @flat_agent_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2237,6 +2445,8 @@ define amdgpu_kernel void @flat_agent_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2352,6 +2562,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_agent_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2369,6 +2582,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2388,6 +2605,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2423,6 +2644,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2438,6 +2661,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2574,6 +2799,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_agent_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2591,6 +2819,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2610,6 +2842,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2645,6 +2881,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2660,6 +2898,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2796,6 +3036,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2821,6 +3064,10 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2846,6 +3093,10 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2896,6 +3147,8 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2910,6 +3163,8 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3019,6 +3274,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3046,6 +3304,10 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3075,6 +3337,10 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3130,6 +3396,8 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3146,6 +3414,8 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3273,6 +3543,9 @@ entry: define amdgpu_kernel void @flat_agent_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3299,6 +3572,10 @@ define amdgpu_kernel void @flat_agent_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3326,6 +3603,10 @@ define amdgpu_kernel void @flat_agent_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3379,6 +3660,8 @@ define amdgpu_kernel void @flat_agent_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3394,6 +3677,8 @@ define amdgpu_kernel void @flat_agent_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3520,6 +3805,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3548,6 +3836,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3579,6 +3871,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3637,6 +3933,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3654,6 +3952,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3798,6 +4098,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3826,6 +4129,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3857,6 +4164,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3915,6 +4226,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3932,6 +4245,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4076,6 +4391,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4103,6 +4421,10 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4132,6 +4454,10 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4187,6 +4513,8 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4203,6 +4531,8 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4330,6 +4660,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4357,6 +4690,10 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4386,6 +4723,10 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4441,6 +4782,8 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4457,6 +4800,8 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4584,6 +4929,9 @@ entry: define amdgpu_kernel void @flat_agent_release_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4612,6 +4960,10 @@ define amdgpu_kernel void @flat_agent_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4643,6 +4995,10 @@ define amdgpu_kernel void @flat_agent_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4701,6 +5057,8 @@ define amdgpu_kernel void @flat_agent_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4718,6 +5076,8 @@ define amdgpu_kernel void @flat_agent_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4862,6 +5222,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4890,6 +5253,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4921,6 +5288,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4979,6 +5350,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4996,6 +5369,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5140,6 +5515,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5168,6 +5546,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5199,6 +5581,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5257,6 +5643,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5274,6 +5662,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5418,6 +5808,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5446,6 +5839,10 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5477,6 +5874,10 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5535,6 +5936,8 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5552,6 +5955,8 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5696,6 +6101,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5724,6 +6132,10 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5755,6 +6167,10 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5813,6 +6229,8 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5830,6 +6248,8 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5974,6 +6394,9 @@ entry: define amdgpu_kernel void @flat_agent_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6002,6 +6425,10 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6033,6 +6460,10 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6091,6 +6522,8 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6108,6 +6541,8 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6252,6 +6687,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6280,6 +6718,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6311,6 +6753,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6369,6 +6815,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6386,6 +6834,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6530,6 +6980,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6558,6 +7011,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6589,6 +7046,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6647,6 +7108,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6664,6 +7127,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6808,6 +7273,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6837,6 +7305,10 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6866,6 +7338,10 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6924,6 +7400,8 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6941,6 +7419,8 @@ define amdgpu_kernel void @flat_agent_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7077,6 +7557,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7107,6 +7590,10 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7138,6 +7625,10 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7198,6 +7689,8 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7216,6 +7709,8 @@ define amdgpu_kernel void @flat_agent_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7361,6 +7856,9 @@ entry: define amdgpu_kernel void @flat_agent_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7391,6 +7889,10 @@ define amdgpu_kernel void @flat_agent_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7422,6 +7924,10 @@ define amdgpu_kernel void @flat_agent_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7483,6 +7989,8 @@ define amdgpu_kernel void @flat_agent_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7501,6 +8009,8 @@ define amdgpu_kernel void @flat_agent_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7654,6 +8164,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7685,6 +8198,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7718,6 +8235,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7781,6 +8302,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7800,6 +8323,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7966,6 +8491,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7997,6 +8525,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8030,6 +8562,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8093,6 +8629,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8112,6 +8650,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8278,6 +8818,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8308,6 +8851,10 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8339,6 +8886,10 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8399,6 +8950,8 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8417,6 +8970,8 @@ define amdgpu_kernel void @flat_agent_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8566,6 +9121,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8596,6 +9154,10 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8627,6 +9189,10 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8687,6 +9253,8 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8705,6 +9273,8 @@ define amdgpu_kernel void @flat_agent_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8850,6 +9420,9 @@ entry: define amdgpu_kernel void @flat_agent_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8881,6 +9454,10 @@ define amdgpu_kernel void @flat_agent_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8914,6 +9491,10 @@ define amdgpu_kernel void @flat_agent_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8977,6 +9558,8 @@ define amdgpu_kernel void @flat_agent_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8996,6 +9579,8 @@ define amdgpu_kernel void @flat_agent_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9162,6 +9747,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9193,6 +9781,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9226,6 +9818,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9289,6 +9885,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9308,6 +9906,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9474,6 +10074,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9505,6 +10108,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9538,6 +10145,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9601,6 +10212,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9620,6 +10233,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9786,6 +10401,9 @@ entry: define amdgpu_kernel void @flat_agent_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9817,6 +10435,10 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9850,6 +10472,10 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9913,6 +10539,8 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9932,6 +10560,8 @@ define amdgpu_kernel void @flat_agent_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10098,6 +10728,9 @@ entry: define amdgpu_kernel void @flat_agent_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10129,6 +10762,10 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10162,6 +10799,10 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10225,6 +10866,8 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10244,6 +10887,8 @@ define amdgpu_kernel void @flat_agent_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10406,6 +11051,9 @@ entry: define amdgpu_kernel void @flat_agent_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10437,6 +11085,10 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10470,6 +11122,10 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10533,6 +11189,8 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10552,6 +11210,8 @@ define amdgpu_kernel void @flat_agent_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10718,6 +11378,9 @@ entry: define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10749,6 +11412,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10782,6 +11449,10 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10845,6 +11516,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10864,6 +11537,8 @@ define amdgpu_kernel void @flat_agent_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -11030,6 +11705,9 @@ entry: define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -11061,6 +11739,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -11094,6 +11776,10 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -11157,6 +11843,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -11176,6 +11864,8 @@ define amdgpu_kernel void @flat_agent_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -11342,6 +12032,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_unordered_load( ; GFX7-LABEL: flat_agent_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11356,6 +12049,10 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_load( ; ; GFX10-WGP-LABEL: flat_agent_one_as_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11370,6 +12067,10 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_load( ; ; GFX10-CU-LABEL: flat_agent_one_as_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11398,6 +12099,8 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11410,6 +12113,8 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11509,6 +12214,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_load( ; GFX7-LABEL: flat_agent_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11523,6 +12231,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_load( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11537,6 +12249,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_load( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11565,6 +12281,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11577,6 +12295,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11676,6 +12396,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_load( ; GFX7-LABEL: flat_agent_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11692,6 +12415,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_load( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11709,6 +12436,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_load( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11741,6 +12472,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11755,6 +12488,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11868,6 +12603,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_load( ; GFX7-LABEL: flat_agent_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11885,6 +12623,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11904,6 +12646,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_load( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11939,6 +12685,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11954,6 +12702,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12086,6 +12836,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_unordered_store( ; GFX7-LABEL: flat_agent_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12097,6 +12850,10 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_store( ; ; GFX10-WGP-LABEL: flat_agent_one_as_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12108,6 +12865,10 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_store( ; ; GFX10-CU-LABEL: flat_agent_one_as_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12130,6 +12891,8 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12140,6 +12903,8 @@ define amdgpu_kernel void @flat_agent_one_as_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12220,6 +12985,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_store( ; GFX7-LABEL: flat_agent_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12231,6 +12999,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_store( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12242,6 +13014,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_store( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12264,6 +13040,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12274,6 +13052,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12354,6 +13134,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_store( ; GFX7-LABEL: flat_agent_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12366,6 +13149,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_store( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12379,6 +13166,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_store( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12404,6 +13195,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12415,6 +13208,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12512,6 +13307,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_store( ; GFX7-LABEL: flat_agent_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12524,6 +13322,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12537,6 +13339,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_store( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12562,6 +13368,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12573,6 +13381,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12670,6 +13480,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12681,6 +13494,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12692,6 +13509,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12714,6 +13535,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12724,6 +13547,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12804,6 +13629,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12817,6 +13645,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12831,6 +13663,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12857,6 +13693,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12869,6 +13707,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12965,6 +13805,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12977,6 +13820,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12990,6 +13837,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13015,6 +13866,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13026,6 +13879,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13123,6 +13978,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13137,6 +13995,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13153,6 +14015,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13182,6 +14048,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13195,6 +14063,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13308,6 +14178,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13322,6 +14195,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13338,6 +14215,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13367,6 +14248,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13380,6 +14263,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13493,6 +14378,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13510,6 +14398,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13528,6 +14420,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13562,6 +14458,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13577,6 +14475,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13697,6 +14597,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13715,6 +14618,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13735,6 +14642,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13772,6 +14683,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13788,6 +14701,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13929,6 +14844,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_agent_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13947,6 +14865,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13967,6 +14889,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -14004,6 +14930,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -14020,6 +14948,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -14161,6 +15091,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14186,6 +15119,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14211,6 +15148,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14261,6 +15202,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14275,6 +15218,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14384,6 +15329,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14411,6 +15359,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14439,6 +15391,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14493,6 +15449,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14509,6 +15467,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14634,6 +15594,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14660,6 +15623,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14687,6 +15654,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14740,6 +15711,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14755,6 +15728,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14881,6 +15856,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14909,6 +15887,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14939,6 +15921,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14996,6 +15982,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15013,6 +16001,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15155,6 +16145,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15183,6 +16176,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15213,6 +16210,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15270,6 +16271,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15287,6 +16290,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15429,6 +16434,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15456,6 +16464,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15484,6 +16496,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15538,6 +16554,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15554,6 +16572,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15679,6 +16699,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15706,6 +16729,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15734,6 +16761,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15788,6 +16819,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15804,6 +16837,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15929,6 +16964,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15957,6 +16995,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15987,6 +17029,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16044,6 +17090,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16061,6 +17109,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16203,6 +17253,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16231,6 +17284,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16261,6 +17318,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16318,6 +17379,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16335,6 +17398,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16477,6 +17542,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16505,6 +17573,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16535,6 +17607,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16592,6 +17668,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16609,6 +17687,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16751,6 +17831,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16779,6 +17862,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16809,6 +17896,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16866,6 +17957,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16883,6 +17976,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17025,6 +18120,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17053,6 +18151,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17083,6 +18185,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17140,6 +18246,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17157,6 +18265,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17299,6 +18409,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17327,6 +18440,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17357,6 +18474,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17414,6 +18535,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17431,6 +18554,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17573,6 +18698,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17601,6 +18729,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17631,6 +18763,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17688,6 +18824,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17705,6 +18843,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17847,6 +18987,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17875,6 +19018,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17905,6 +19052,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17962,6 +19113,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17979,6 +19132,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18121,6 +19276,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18150,6 +19308,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18179,6 +19341,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18237,6 +19403,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18254,6 +19422,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18390,6 +19560,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18421,6 +19594,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18453,6 +19630,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18515,6 +19696,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18534,6 +19717,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18684,6 +19869,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18714,6 +19902,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18745,6 +19937,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18806,6 +20002,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18824,6 +20022,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18977,6 +20177,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19009,6 +20212,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19043,6 +20250,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19108,6 +20319,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19128,6 +20341,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19299,6 +20514,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19331,6 +20549,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19365,6 +20587,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19430,6 +20656,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19450,6 +20678,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19621,6 +20851,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19652,6 +20885,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19684,6 +20921,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19746,6 +20987,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19765,6 +21008,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19919,6 +21164,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19950,6 +21198,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19982,6 +21234,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20044,6 +21300,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20063,6 +21321,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20213,6 +21473,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20245,6 +21508,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20279,6 +21546,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20344,6 +21615,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20364,6 +21637,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20535,6 +21810,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20567,6 +21845,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20601,6 +21883,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20666,6 +21952,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20686,6 +21974,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20857,6 +22147,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20889,6 +22182,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20923,6 +22220,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20988,6 +22289,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21008,6 +22311,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21179,6 +22484,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21211,6 +22519,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21245,6 +22557,10 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21310,6 +22626,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21330,6 +22648,8 @@ define amdgpu_kernel void @flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21501,6 +22821,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21533,6 +22856,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21567,6 +22894,10 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21632,6 +22963,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21652,6 +22985,8 @@ define amdgpu_kernel void @flat_agent_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21819,6 +23154,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21851,6 +23189,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21885,6 +23227,10 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21950,6 +23296,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21970,6 +23318,8 @@ define amdgpu_kernel void @flat_agent_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22141,6 +23491,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -22173,6 +23526,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22207,6 +23564,10 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22272,6 +23633,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22292,6 +23655,8 @@ define amdgpu_kernel void @flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22463,6 +23828,9 @@ entry: define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -22495,6 +23863,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22529,6 +23901,10 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22594,6 +23970,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22614,6 +23992,8 @@ define amdgpu_kernel void @flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_agent_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-nontemporal.ll index 30c0a322d7ddc..3c24c36ec547d 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-nontemporal.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-nontemporal.ll @@ -15,6 +15,9 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; GFX7-LABEL: flat_nontemporal_load_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -29,6 +32,10 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; ; GFX10-WGP-LABEL: flat_nontemporal_load_0: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -43,6 +50,10 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; ; GFX10-CU-LABEL: flat_nontemporal_load_0: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -71,6 +82,8 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_nontemporal_load_0: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -83,6 +96,8 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; ; GFX90A-TGSPLIT-LABEL: flat_nontemporal_load_0: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -182,6 +197,9 @@ entry: define amdgpu_kernel void @flat_nontemporal_load_1( ; GFX7-LABEL: flat_nontemporal_load_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[4:5], 0x2 @@ -211,6 +229,10 @@ define amdgpu_kernel void @flat_nontemporal_load_1( ; ; GFX10-WGP-LABEL: flat_nontemporal_load_1: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_nop 0 @@ -240,6 +262,10 @@ define amdgpu_kernel void @flat_nontemporal_load_1( ; ; GFX10-CU-LABEL: flat_nontemporal_load_1: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_nop 0 @@ -298,6 +324,8 @@ define amdgpu_kernel void @flat_nontemporal_load_1( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_nontemporal_load_1: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_nop 0 @@ -329,6 +357,8 @@ define amdgpu_kernel void @flat_nontemporal_load_1( ; ; GFX90A-TGSPLIT-LABEL: flat_nontemporal_load_1: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90A-TGSPLIT-NEXT: s_nop 0 @@ -537,6 +567,9 @@ entry: define amdgpu_kernel void @flat_nontemporal_store_0( ; GFX7-LABEL: flat_nontemporal_store_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -551,6 +584,10 @@ define amdgpu_kernel void @flat_nontemporal_store_0( ; ; GFX10-WGP-LABEL: flat_nontemporal_store_0: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -565,6 +602,10 @@ define amdgpu_kernel void @flat_nontemporal_store_0( ; ; GFX10-CU-LABEL: flat_nontemporal_store_0: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -593,6 +634,8 @@ define amdgpu_kernel void @flat_nontemporal_store_0( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_nontemporal_store_0: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -605,6 +648,8 @@ define amdgpu_kernel void @flat_nontemporal_store_0( ; ; GFX90A-TGSPLIT-LABEL: flat_nontemporal_store_0: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -704,6 +749,9 @@ entry: define amdgpu_kernel void @flat_nontemporal_store_1( ; GFX7-LABEL: flat_nontemporal_store_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -732,6 +780,10 @@ define amdgpu_kernel void @flat_nontemporal_store_1( ; ; GFX10-WGP-LABEL: flat_nontemporal_store_1: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -759,6 +811,10 @@ define amdgpu_kernel void @flat_nontemporal_store_1( ; ; GFX10-CU-LABEL: flat_nontemporal_store_1: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -814,6 +870,8 @@ define amdgpu_kernel void @flat_nontemporal_store_1( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_nontemporal_store_1: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -843,6 +901,8 @@ define amdgpu_kernel void @flat_nontemporal_store_1( ; ; GFX90A-TGSPLIT-LABEL: flat_nontemporal_store_1: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1047,6 +1107,9 @@ entry: define amdgpu_kernel void @flat_nontemporal_volatile_load( ; GFX7-LABEL: flat_nontemporal_volatile_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1062,6 +1125,10 @@ define amdgpu_kernel void @flat_nontemporal_volatile_load( ; ; GFX10-WGP-LABEL: flat_nontemporal_volatile_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1077,6 +1144,10 @@ define amdgpu_kernel void @flat_nontemporal_volatile_load( ; ; GFX10-CU-LABEL: flat_nontemporal_volatile_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1107,6 +1178,8 @@ define amdgpu_kernel void @flat_nontemporal_volatile_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_nontemporal_volatile_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1120,6 +1193,8 @@ define amdgpu_kernel void @flat_nontemporal_volatile_load( ; ; GFX90A-TGSPLIT-LABEL: flat_nontemporal_volatile_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-singlethread.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-singlethread.ll index b80dfaea01653..b88a10ab24a98 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-singlethread.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-singlethread.ll @@ -15,6 +15,9 @@ define amdgpu_kernel void @flat_singlethread_unordered_load( ; GFX7-LABEL: flat_singlethread_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -29,6 +32,10 @@ define amdgpu_kernel void @flat_singlethread_unordered_load( ; ; GFX10-WGP-LABEL: flat_singlethread_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -43,6 +50,10 @@ define amdgpu_kernel void @flat_singlethread_unordered_load( ; ; GFX10-CU-LABEL: flat_singlethread_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -71,6 +82,8 @@ define amdgpu_kernel void @flat_singlethread_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -83,6 +96,8 @@ define amdgpu_kernel void @flat_singlethread_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -182,6 +197,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_load( ; GFX7-LABEL: flat_singlethread_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -196,6 +214,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_load( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -210,6 +232,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_load( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -238,6 +264,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -250,6 +278,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -349,6 +379,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_load( ; GFX7-LABEL: flat_singlethread_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -363,6 +396,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_load( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -377,6 +414,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_load( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -405,6 +446,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -417,6 +460,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -516,6 +561,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_load( ; GFX7-LABEL: flat_singlethread_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -530,6 +578,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -544,6 +596,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_load( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -572,6 +628,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -584,6 +642,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -683,6 +743,9 @@ entry: define amdgpu_kernel void @flat_singlethread_unordered_store( ; GFX7-LABEL: flat_singlethread_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -694,6 +757,10 @@ define amdgpu_kernel void @flat_singlethread_unordered_store( ; ; GFX10-WGP-LABEL: flat_singlethread_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -705,6 +772,10 @@ define amdgpu_kernel void @flat_singlethread_unordered_store( ; ; GFX10-CU-LABEL: flat_singlethread_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -727,6 +798,8 @@ define amdgpu_kernel void @flat_singlethread_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -737,6 +810,8 @@ define amdgpu_kernel void @flat_singlethread_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -817,6 +892,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_store( ; GFX7-LABEL: flat_singlethread_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -828,6 +906,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_store( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -839,6 +921,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_store( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -861,6 +947,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -871,6 +959,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -951,6 +1041,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_store( ; GFX7-LABEL: flat_singlethread_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -962,6 +1055,10 @@ define amdgpu_kernel void @flat_singlethread_release_store( ; ; GFX10-WGP-LABEL: flat_singlethread_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -973,6 +1070,10 @@ define amdgpu_kernel void @flat_singlethread_release_store( ; ; GFX10-CU-LABEL: flat_singlethread_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -995,6 +1096,8 @@ define amdgpu_kernel void @flat_singlethread_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1005,6 +1108,8 @@ define amdgpu_kernel void @flat_singlethread_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1085,6 +1190,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_store( ; GFX7-LABEL: flat_singlethread_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1096,6 +1204,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1107,6 +1219,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_store( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1129,6 +1245,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1139,6 +1257,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1219,6 +1339,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_atomicrmw( ; GFX7-LABEL: flat_singlethread_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1230,6 +1353,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1241,6 +1368,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1263,6 +1394,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1273,6 +1406,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1353,6 +1488,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_atomicrmw( ; GFX7-LABEL: flat_singlethread_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1364,6 +1502,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1375,6 +1517,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1397,6 +1543,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1407,6 +1555,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1487,6 +1637,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_atomicrmw( ; GFX7-LABEL: flat_singlethread_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1498,6 +1651,10 @@ define amdgpu_kernel void @flat_singlethread_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1509,6 +1666,10 @@ define amdgpu_kernel void @flat_singlethread_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1531,6 +1692,8 @@ define amdgpu_kernel void @flat_singlethread_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1541,6 +1704,8 @@ define amdgpu_kernel void @flat_singlethread_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1621,6 +1786,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_atomicrmw( ; GFX7-LABEL: flat_singlethread_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1632,6 +1800,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1643,6 +1815,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1665,6 +1841,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1675,6 +1853,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1755,6 +1935,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_atomicrmw( ; GFX7-LABEL: flat_singlethread_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1766,6 +1949,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1777,6 +1964,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1799,6 +1990,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1809,6 +2002,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1889,6 +2084,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_singlethread_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1904,6 +2102,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1919,6 +2121,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1949,6 +2155,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1962,6 +2170,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2068,6 +2278,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_singlethread_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2083,6 +2296,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2098,6 +2315,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2128,6 +2349,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2141,6 +2364,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2247,6 +2472,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_singlethread_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2262,6 +2490,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2277,6 +2509,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2307,6 +2543,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2320,6 +2558,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2426,6 +2666,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2451,6 +2694,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2476,6 +2723,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2526,6 +2777,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2540,6 +2793,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2649,6 +2904,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2674,6 +2932,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2699,6 +2961,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2749,6 +3015,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2763,6 +3031,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2872,6 +3142,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2897,6 +3170,10 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2922,6 +3199,10 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2972,6 +3253,8 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2986,6 +3269,8 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3095,6 +3380,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3120,6 +3408,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3145,6 +3437,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3195,6 +3491,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3209,6 +3507,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3318,6 +3618,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3343,6 +3646,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3368,6 +3675,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3418,6 +3729,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3432,6 +3745,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3541,6 +3856,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3566,6 +3884,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3591,6 +3913,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3641,6 +3967,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3655,6 +3983,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3764,6 +4094,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3789,6 +4122,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3814,6 +4151,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3864,6 +4205,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3878,6 +4221,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3987,6 +4332,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4012,6 +4360,10 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4037,6 +4389,10 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4087,6 +4443,8 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4101,6 +4459,8 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4210,6 +4570,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4235,6 +4598,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4260,6 +4627,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4310,6 +4681,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4324,6 +4697,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4433,6 +4808,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4458,6 +4836,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4483,6 +4865,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4533,6 +4919,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4547,6 +4935,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4656,6 +5046,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4681,6 +5074,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4706,6 +5103,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4756,6 +5157,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4770,6 +5173,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4879,6 +5284,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4904,6 +5312,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4929,6 +5341,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4979,6 +5395,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4993,6 +5411,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5102,6 +5522,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5127,6 +5550,10 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5152,6 +5579,10 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5202,6 +5633,8 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5216,6 +5649,8 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5325,6 +5760,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5350,6 +5788,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5375,6 +5817,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5425,6 +5871,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5439,6 +5887,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5548,6 +5998,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5573,6 +6026,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5598,6 +6055,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5648,6 +6109,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5662,6 +6125,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5771,6 +6236,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -5800,6 +6268,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5829,6 +6301,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5887,6 +6363,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5904,6 +6382,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6040,6 +6520,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6069,6 +6552,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6098,6 +6585,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6156,6 +6647,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6173,6 +6666,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6309,6 +6804,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6338,6 +6836,10 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6367,6 +6869,10 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6425,6 +6931,8 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6442,6 +6950,8 @@ define amdgpu_kernel void @flat_singlethread_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6578,6 +7088,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6607,6 +7120,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6636,6 +7153,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6694,6 +7215,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6711,6 +7234,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6847,6 +7372,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6876,6 +7404,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6905,6 +7437,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6963,6 +7499,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6980,6 +7518,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7116,6 +7656,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7145,6 +7688,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7174,6 +7721,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7232,6 +7783,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7249,6 +7802,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7385,6 +7940,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7414,6 +7972,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7443,6 +8005,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7501,6 +8067,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7518,6 +8086,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7654,6 +8224,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7683,6 +8256,10 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7712,6 +8289,10 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7770,6 +8351,8 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7787,6 +8370,8 @@ define amdgpu_kernel void @flat_singlethread_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7923,6 +8508,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7952,6 +8540,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7981,6 +8573,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8039,6 +8635,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8056,6 +8654,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8192,6 +8792,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8221,6 +8824,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8250,6 +8857,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8308,6 +8919,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8325,6 +8938,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8461,6 +9076,9 @@ entry: define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8490,6 +9108,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8519,6 +9141,10 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8577,6 +9203,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8594,6 +9222,8 @@ define amdgpu_kernel void @flat_singlethread_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8730,6 +9360,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8759,6 +9392,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8788,6 +9425,10 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8846,6 +9487,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8863,6 +9506,8 @@ define amdgpu_kernel void @flat_singlethread_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8999,6 +9644,9 @@ entry: define amdgpu_kernel void @flat_singlethread_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9028,6 +9676,10 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9057,6 +9709,10 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9115,6 +9771,8 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9132,6 +9790,8 @@ define amdgpu_kernel void @flat_singlethread_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9268,6 +9928,9 @@ entry: define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9297,6 +9960,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9326,6 +9993,10 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9384,6 +10055,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9401,6 +10074,8 @@ define amdgpu_kernel void @flat_singlethread_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9537,6 +10212,9 @@ entry: define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9566,6 +10244,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9595,6 +10277,10 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9653,6 +10339,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9670,6 +10358,8 @@ define amdgpu_kernel void @flat_singlethread_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9806,6 +10496,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_unordered_load( ; GFX7-LABEL: flat_singlethread_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9820,6 +10513,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_load( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -9834,6 +10531,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_load( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -9862,6 +10563,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -9874,6 +10577,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -9973,6 +10678,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_load( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9987,6 +10695,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_load( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10001,6 +10713,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_load( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10029,6 +10745,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10041,6 +10759,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10140,6 +10860,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_load( ; GFX7-LABEL: flat_singlethread_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10154,6 +10877,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_load( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10168,6 +10895,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_load( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10196,6 +10927,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10208,6 +10941,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10307,6 +11042,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_load( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10321,6 +11059,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10335,6 +11077,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_load( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10363,6 +11109,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10375,6 +11123,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10474,6 +11224,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_unordered_store( ; GFX7-LABEL: flat_singlethread_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10485,6 +11238,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_store( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10496,6 +11253,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_store( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10518,6 +11279,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10528,6 +11291,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10608,6 +11373,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_store( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10619,6 +11387,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_store( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10630,6 +11402,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_store( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10652,6 +11428,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10662,6 +11440,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10742,6 +11522,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_store( ; GFX7-LABEL: flat_singlethread_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10753,6 +11536,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_store( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10764,6 +11551,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_store( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10786,6 +11577,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10796,6 +11589,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10876,6 +11671,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_store( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10887,6 +11685,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10898,6 +11700,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_store( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10920,6 +11726,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10930,6 +11738,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11010,6 +11820,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11021,6 +11834,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11032,6 +11849,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11054,6 +11875,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11064,6 +11887,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11144,6 +11969,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11155,6 +11983,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11166,6 +11998,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11188,6 +12024,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11198,6 +12036,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11278,6 +12118,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11289,6 +12132,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11300,6 +12147,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11322,6 +12173,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11332,6 +12185,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11412,6 +12267,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11423,6 +12281,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11434,6 +12296,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11456,6 +12322,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11466,6 +12334,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11546,6 +12416,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11557,6 +12430,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11568,6 +12445,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11590,6 +12471,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11600,6 +12483,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11680,6 +12565,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11695,6 +12583,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11710,6 +12602,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11740,6 +12636,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11753,6 +12651,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11859,6 +12759,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11874,6 +12777,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11889,6 +12796,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11919,6 +12830,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11932,6 +12845,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12038,6 +12953,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12053,6 +12971,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12068,6 +12990,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12098,6 +13024,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12111,6 +13039,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12217,6 +13147,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12242,6 +13175,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12267,6 +13204,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12317,6 +13258,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12331,6 +13274,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12440,6 +13385,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12465,6 +13413,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12490,6 +13442,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12540,6 +13496,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12554,6 +13512,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12663,6 +13623,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12688,6 +13651,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12713,6 +13680,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12763,6 +13734,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12777,6 +13750,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12886,6 +13861,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12911,6 +13889,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12936,6 +13918,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12986,6 +13972,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13000,6 +13988,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13109,6 +14099,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13134,6 +14127,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13159,6 +14156,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13209,6 +14210,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13223,6 +14226,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13332,6 +14337,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13357,6 +14365,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13382,6 +14394,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13432,6 +14448,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13446,6 +14464,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13555,6 +14575,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13580,6 +14603,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13605,6 +14632,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13655,6 +14686,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13669,6 +14702,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13778,6 +14813,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13803,6 +14841,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13828,6 +14870,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13878,6 +14924,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13892,6 +14940,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14001,6 +15051,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14026,6 +15079,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14051,6 +15108,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14101,6 +15162,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14115,6 +15178,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14224,6 +15289,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14249,6 +15317,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14274,6 +15346,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14324,6 +15400,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14338,6 +15416,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14447,6 +15527,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14472,6 +15555,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14497,6 +15584,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14547,6 +15638,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14561,6 +15654,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14670,6 +15765,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14695,6 +15793,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14720,6 +15822,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14770,6 +15876,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14784,6 +15892,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14893,6 +16003,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14918,6 +16031,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14943,6 +16060,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14993,6 +16114,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15007,6 +16130,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15116,6 +16241,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15141,6 +16269,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15166,6 +16298,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15216,6 +16352,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15230,6 +16368,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15339,6 +16479,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15364,6 +16507,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15389,6 +16536,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15439,6 +16590,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15453,6 +16606,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15562,6 +16717,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15591,6 +16749,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_ret_cmpx ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15620,6 +16782,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_ret_cmpx ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15678,6 +16844,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_ret_cmpx ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15695,6 +16863,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_monotonic_ret_cmpx ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15831,6 +17001,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15860,6 +17033,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_ret_cmpxch ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15889,6 +17066,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_ret_cmpxch ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15947,6 +17128,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_ret_cmpxch ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15964,6 +17147,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_monotonic_ret_cmpxch ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16100,6 +17285,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16129,6 +17317,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_ret_cmpxch ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16158,6 +17350,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_ret_cmpxch ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16216,6 +17412,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_ret_cmpxch ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16233,6 +17431,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_monotonic_ret_cmpxch ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16369,6 +17569,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16398,6 +17601,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxch ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16427,6 +17634,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxch ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16485,6 +17696,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxch ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16502,6 +17715,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxch ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16638,6 +17853,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16667,6 +17885,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxch ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16696,6 +17918,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxch ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16754,6 +17980,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxch ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16771,6 +17999,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxch ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16907,6 +18137,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16936,6 +18169,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_ret_cmpxch ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16965,6 +18202,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_ret_cmpxch ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17023,6 +18264,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_ret_cmpxch ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17040,6 +18283,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_acquire_ret_cmpxch ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17176,6 +18421,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17205,6 +18453,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17234,6 +18486,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17292,6 +18548,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17309,6 +18567,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17445,6 +18705,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17474,6 +18737,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17503,6 +18770,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17561,6 +18832,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17578,6 +18851,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17714,6 +18989,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17743,6 +19021,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17772,6 +19054,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17830,6 +19116,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17847,6 +19135,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17983,6 +19273,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18012,6 +19305,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18041,6 +19338,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18099,6 +19400,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18116,6 +19419,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18252,6 +19557,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18281,6 +19589,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxch ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18310,6 +19622,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxch ; ; GFX10-CU-LABEL: flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18368,6 +19684,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxch ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18385,6 +19703,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxch ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18521,6 +19841,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18550,6 +19873,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18579,6 +19906,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18637,6 +19968,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18654,6 +19987,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18790,6 +20125,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18819,6 +20157,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18848,6 +20190,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18906,6 +20252,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18923,6 +20271,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19059,6 +20409,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19088,6 +20441,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19117,6 +20474,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19175,6 +20536,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19192,6 +20555,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19328,6 +20693,9 @@ entry: define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19357,6 +20725,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19386,6 +20758,10 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19444,6 +20820,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19461,6 +20839,8 @@ define amdgpu_kernel void @flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-system.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-system.ll index 1ec942ea5f47b..919fc3e8f4e4f 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-system.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-system.ll @@ -15,6 +15,9 @@ define amdgpu_kernel void @flat_system_unordered_load( ; GFX7-LABEL: flat_system_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -29,6 +32,10 @@ define amdgpu_kernel void @flat_system_unordered_load( ; ; GFX10-WGP-LABEL: flat_system_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -43,6 +50,10 @@ define amdgpu_kernel void @flat_system_unordered_load( ; ; GFX10-CU-LABEL: flat_system_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -71,6 +82,8 @@ define amdgpu_kernel void @flat_system_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -83,6 +96,8 @@ define amdgpu_kernel void @flat_system_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -182,6 +197,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_load( ; GFX7-LABEL: flat_system_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -196,6 +214,10 @@ define amdgpu_kernel void @flat_system_monotonic_load( ; ; GFX10-WGP-LABEL: flat_system_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -210,6 +232,10 @@ define amdgpu_kernel void @flat_system_monotonic_load( ; ; GFX10-CU-LABEL: flat_system_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -238,6 +264,8 @@ define amdgpu_kernel void @flat_system_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -250,6 +278,8 @@ define amdgpu_kernel void @flat_system_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -349,6 +379,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_load( ; GFX7-LABEL: flat_system_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -364,6 +397,10 @@ define amdgpu_kernel void @flat_system_acquire_load( ; ; GFX10-WGP-LABEL: flat_system_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -380,6 +417,10 @@ define amdgpu_kernel void @flat_system_acquire_load( ; ; GFX10-CU-LABEL: flat_system_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -410,6 +451,8 @@ define amdgpu_kernel void @flat_system_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -424,6 +467,8 @@ define amdgpu_kernel void @flat_system_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -533,6 +578,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_load( ; GFX7-LABEL: flat_system_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -549,6 +597,10 @@ define amdgpu_kernel void @flat_system_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -567,6 +619,10 @@ define amdgpu_kernel void @flat_system_seq_cst_load( ; ; GFX10-CU-LABEL: flat_system_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -600,6 +656,8 @@ define amdgpu_kernel void @flat_system_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -615,6 +673,8 @@ define amdgpu_kernel void @flat_system_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -743,6 +803,9 @@ entry: define amdgpu_kernel void @flat_system_unordered_store( ; GFX7-LABEL: flat_system_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -754,6 +817,10 @@ define amdgpu_kernel void @flat_system_unordered_store( ; ; GFX10-WGP-LABEL: flat_system_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -765,6 +832,10 @@ define amdgpu_kernel void @flat_system_unordered_store( ; ; GFX10-CU-LABEL: flat_system_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -787,6 +858,8 @@ define amdgpu_kernel void @flat_system_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -797,6 +870,8 @@ define amdgpu_kernel void @flat_system_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -877,6 +952,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_store( ; GFX7-LABEL: flat_system_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -888,6 +966,10 @@ define amdgpu_kernel void @flat_system_monotonic_store( ; ; GFX10-WGP-LABEL: flat_system_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -899,6 +981,10 @@ define amdgpu_kernel void @flat_system_monotonic_store( ; ; GFX10-CU-LABEL: flat_system_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -921,6 +1007,8 @@ define amdgpu_kernel void @flat_system_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -931,6 +1019,8 @@ define amdgpu_kernel void @flat_system_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1011,6 +1101,9 @@ entry: define amdgpu_kernel void @flat_system_release_store( ; GFX7-LABEL: flat_system_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1023,6 +1116,10 @@ define amdgpu_kernel void @flat_system_release_store( ; ; GFX10-WGP-LABEL: flat_system_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1036,6 +1133,10 @@ define amdgpu_kernel void @flat_system_release_store( ; ; GFX10-CU-LABEL: flat_system_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1061,6 +1162,8 @@ define amdgpu_kernel void @flat_system_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1073,6 +1176,8 @@ define amdgpu_kernel void @flat_system_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1173,6 +1278,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_store( ; GFX7-LABEL: flat_system_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1185,6 +1293,10 @@ define amdgpu_kernel void @flat_system_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1198,6 +1310,10 @@ define amdgpu_kernel void @flat_system_seq_cst_store( ; ; GFX10-CU-LABEL: flat_system_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1223,6 +1339,8 @@ define amdgpu_kernel void @flat_system_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1235,6 +1353,8 @@ define amdgpu_kernel void @flat_system_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1335,6 +1455,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_atomicrmw( ; GFX7-LABEL: flat_system_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1346,6 +1469,10 @@ define amdgpu_kernel void @flat_system_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1357,6 +1484,10 @@ define amdgpu_kernel void @flat_system_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1379,6 +1510,8 @@ define amdgpu_kernel void @flat_system_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1389,6 +1522,8 @@ define amdgpu_kernel void @flat_system_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1469,6 +1604,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_atomicrmw( ; GFX7-LABEL: flat_system_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1482,6 +1620,10 @@ define amdgpu_kernel void @flat_system_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1497,6 +1639,10 @@ define amdgpu_kernel void @flat_system_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1524,6 +1670,8 @@ define amdgpu_kernel void @flat_system_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1537,6 +1685,8 @@ define amdgpu_kernel void @flat_system_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1636,6 +1786,9 @@ entry: define amdgpu_kernel void @flat_system_release_atomicrmw( ; GFX7-LABEL: flat_system_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1648,6 +1801,10 @@ define amdgpu_kernel void @flat_system_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1661,6 +1818,10 @@ define amdgpu_kernel void @flat_system_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1686,6 +1847,8 @@ define amdgpu_kernel void @flat_system_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1698,6 +1861,8 @@ define amdgpu_kernel void @flat_system_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1798,6 +1963,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_atomicrmw( ; GFX7-LABEL: flat_system_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1812,6 +1980,10 @@ define amdgpu_kernel void @flat_system_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1829,6 +2001,10 @@ define amdgpu_kernel void @flat_system_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1859,6 +2035,8 @@ define amdgpu_kernel void @flat_system_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1874,6 +2052,8 @@ define amdgpu_kernel void @flat_system_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1993,6 +2173,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_atomicrmw( ; GFX7-LABEL: flat_system_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2007,6 +2190,10 @@ define amdgpu_kernel void @flat_system_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2024,6 +2211,10 @@ define amdgpu_kernel void @flat_system_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2054,6 +2245,8 @@ define amdgpu_kernel void @flat_system_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2069,6 +2262,8 @@ define amdgpu_kernel void @flat_system_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2188,6 +2383,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_system_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2204,6 +2402,10 @@ define amdgpu_kernel void @flat_system_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2221,6 +2423,10 @@ define amdgpu_kernel void @flat_system_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2253,6 +2459,8 @@ define amdgpu_kernel void @flat_system_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2268,6 +2476,8 @@ define amdgpu_kernel void @flat_system_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2384,6 +2594,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_system_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2401,6 +2614,10 @@ define amdgpu_kernel void @flat_system_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2420,6 +2637,10 @@ define amdgpu_kernel void @flat_system_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2455,6 +2676,8 @@ define amdgpu_kernel void @flat_system_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2472,6 +2695,8 @@ define amdgpu_kernel void @flat_system_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2612,6 +2837,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_system_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2629,6 +2857,10 @@ define amdgpu_kernel void @flat_system_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2648,6 +2880,10 @@ define amdgpu_kernel void @flat_system_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2683,6 +2919,8 @@ define amdgpu_kernel void @flat_system_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2700,6 +2938,8 @@ define amdgpu_kernel void @flat_system_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2840,6 +3080,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2865,6 +3108,10 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2890,6 +3137,10 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2940,6 +3191,8 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2954,6 +3207,8 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3063,6 +3318,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3090,6 +3348,10 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3119,6 +3381,10 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3174,6 +3440,8 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3191,6 +3459,8 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3319,6 +3589,9 @@ entry: define amdgpu_kernel void @flat_system_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3345,6 +3618,10 @@ define amdgpu_kernel void @flat_system_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3372,6 +3649,10 @@ define amdgpu_kernel void @flat_system_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3425,6 +3706,8 @@ define amdgpu_kernel void @flat_system_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3441,6 +3724,8 @@ define amdgpu_kernel void @flat_system_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3570,6 +3855,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3598,6 +3886,10 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3629,6 +3921,10 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3687,6 +3983,8 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3706,6 +4004,8 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3854,6 +4154,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3882,6 +4185,10 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3913,6 +4220,10 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3971,6 +4282,8 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3990,6 +4303,8 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4138,6 +4453,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_system_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4165,6 +4483,10 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4194,6 +4516,10 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4249,6 +4575,8 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4266,6 +4594,8 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4394,6 +4724,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_system_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4421,6 +4754,10 @@ define amdgpu_kernel void @flat_system_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4450,6 +4787,10 @@ define amdgpu_kernel void @flat_system_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4505,6 +4846,8 @@ define amdgpu_kernel void @flat_system_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4522,6 +4865,8 @@ define amdgpu_kernel void @flat_system_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4650,6 +4995,9 @@ entry: define amdgpu_kernel void @flat_system_release_acquire_cmpxchg( ; GFX7-LABEL: flat_system_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4678,6 +5026,10 @@ define amdgpu_kernel void @flat_system_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4709,6 +5061,10 @@ define amdgpu_kernel void @flat_system_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4767,6 +5123,8 @@ define amdgpu_kernel void @flat_system_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4786,6 +5144,8 @@ define amdgpu_kernel void @flat_system_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4934,6 +5294,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_system_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4962,6 +5325,10 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4993,6 +5360,10 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5051,6 +5422,8 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5070,6 +5443,8 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5218,6 +5593,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_system_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5246,6 +5624,10 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5277,6 +5659,10 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5335,6 +5721,8 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5354,6 +5742,8 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5502,6 +5892,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5530,6 +5923,10 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5561,6 +5958,10 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5619,6 +6020,8 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5638,6 +6041,8 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5786,6 +6191,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5814,6 +6222,10 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5845,6 +6257,10 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5903,6 +6319,8 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5922,6 +6340,8 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6070,6 +6490,9 @@ entry: define amdgpu_kernel void @flat_system_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6098,6 +6521,10 @@ define amdgpu_kernel void @flat_system_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6129,6 +6556,10 @@ define amdgpu_kernel void @flat_system_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6187,6 +6618,8 @@ define amdgpu_kernel void @flat_system_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6206,6 +6639,8 @@ define amdgpu_kernel void @flat_system_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6354,6 +6789,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6382,6 +6820,10 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6413,6 +6855,10 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6471,6 +6917,8 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6490,6 +6938,8 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6638,6 +7088,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6666,6 +7119,10 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6697,6 +7154,10 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -6755,6 +7216,8 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6774,6 +7237,8 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6922,6 +7387,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6951,6 +7419,10 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6980,6 +7452,10 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7038,6 +7514,8 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7055,6 +7533,8 @@ define amdgpu_kernel void @flat_system_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7191,6 +7671,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7221,6 +7704,10 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7252,6 +7739,10 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7312,6 +7803,8 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7331,6 +7824,8 @@ define amdgpu_kernel void @flat_system_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7477,6 +7972,9 @@ entry: define amdgpu_kernel void @flat_system_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7507,6 +8005,10 @@ define amdgpu_kernel void @flat_system_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7538,6 +8040,10 @@ define amdgpu_kernel void @flat_system_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7599,6 +8105,8 @@ define amdgpu_kernel void @flat_system_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7618,6 +8126,8 @@ define amdgpu_kernel void @flat_system_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7774,6 +8284,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7805,6 +8318,10 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7838,6 +8355,10 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7901,6 +8422,8 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7922,6 +8445,8 @@ define amdgpu_kernel void @flat_system_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8092,6 +8617,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8123,6 +8651,10 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8156,6 +8688,10 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8219,6 +8755,8 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8240,6 +8778,8 @@ define amdgpu_kernel void @flat_system_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8410,6 +8950,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8440,6 +8983,10 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8471,6 +9018,10 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8531,6 +9082,8 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8550,6 +9103,8 @@ define amdgpu_kernel void @flat_system_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8700,6 +9255,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8730,6 +9288,10 @@ define amdgpu_kernel void @flat_system_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8761,6 +9323,10 @@ define amdgpu_kernel void @flat_system_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8821,6 +9387,8 @@ define amdgpu_kernel void @flat_system_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8840,6 +9408,8 @@ define amdgpu_kernel void @flat_system_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8986,6 +9556,9 @@ entry: define amdgpu_kernel void @flat_system_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9017,6 +9590,10 @@ define amdgpu_kernel void @flat_system_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9050,6 +9627,10 @@ define amdgpu_kernel void @flat_system_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9113,6 +9694,8 @@ define amdgpu_kernel void @flat_system_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9134,6 +9717,8 @@ define amdgpu_kernel void @flat_system_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9304,6 +9889,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9335,6 +9923,10 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9368,6 +9960,10 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9431,6 +10027,8 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9452,6 +10050,8 @@ define amdgpu_kernel void @flat_system_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9622,6 +10222,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9653,6 +10256,10 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9686,6 +10293,10 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9749,6 +10360,8 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9770,6 +10383,8 @@ define amdgpu_kernel void @flat_system_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9940,6 +10555,9 @@ entry: define amdgpu_kernel void @flat_system_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9971,6 +10589,10 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10004,6 +10626,10 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10067,6 +10693,8 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10088,6 +10716,8 @@ define amdgpu_kernel void @flat_system_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10258,6 +10888,9 @@ entry: define amdgpu_kernel void @flat_system_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10289,6 +10922,10 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10322,6 +10959,10 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10385,6 +11026,8 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10406,6 +11049,8 @@ define amdgpu_kernel void @flat_system_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10572,6 +11217,9 @@ entry: define amdgpu_kernel void @flat_system_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10603,6 +11251,10 @@ define amdgpu_kernel void @flat_system_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10636,6 +11288,10 @@ define amdgpu_kernel void @flat_system_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10699,6 +11355,8 @@ define amdgpu_kernel void @flat_system_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10720,6 +11378,8 @@ define amdgpu_kernel void @flat_system_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -10890,6 +11550,9 @@ entry: define amdgpu_kernel void @flat_system_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10921,6 +11584,10 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -10954,6 +11621,10 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -11017,6 +11688,8 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -11038,6 +11711,8 @@ define amdgpu_kernel void @flat_system_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -11208,6 +11883,9 @@ entry: define amdgpu_kernel void @flat_system_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -11239,6 +11917,10 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -11272,6 +11954,10 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -11335,6 +12021,8 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -11356,6 +12044,8 @@ define amdgpu_kernel void @flat_system_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -11526,6 +12216,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_unordered_load( ; GFX7-LABEL: flat_system_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11540,6 +12233,10 @@ define amdgpu_kernel void @flat_system_one_as_unordered_load( ; ; GFX10-WGP-LABEL: flat_system_one_as_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11554,6 +12251,10 @@ define amdgpu_kernel void @flat_system_one_as_unordered_load( ; ; GFX10-CU-LABEL: flat_system_one_as_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11582,6 +12283,8 @@ define amdgpu_kernel void @flat_system_one_as_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11594,6 +12297,8 @@ define amdgpu_kernel void @flat_system_one_as_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11693,6 +12398,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_load( ; GFX7-LABEL: flat_system_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11707,6 +12415,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_load( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11721,6 +12433,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_load( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11749,6 +12465,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11761,6 +12479,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11860,6 +12580,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_load( ; GFX7-LABEL: flat_system_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11876,6 +12599,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_load( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11893,6 +12620,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_load( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11925,6 +12656,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11940,6 +12673,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12054,6 +12789,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_load( ; GFX7-LABEL: flat_system_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12071,6 +12809,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12090,6 +12832,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_load( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12125,6 +12871,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12141,6 +12889,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12274,6 +13024,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_unordered_store( ; GFX7-LABEL: flat_system_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12285,6 +13038,10 @@ define amdgpu_kernel void @flat_system_one_as_unordered_store( ; ; GFX10-WGP-LABEL: flat_system_one_as_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12296,6 +13053,10 @@ define amdgpu_kernel void @flat_system_one_as_unordered_store( ; ; GFX10-CU-LABEL: flat_system_one_as_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12318,6 +13079,8 @@ define amdgpu_kernel void @flat_system_one_as_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12328,6 +13091,8 @@ define amdgpu_kernel void @flat_system_one_as_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12408,6 +13173,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_store( ; GFX7-LABEL: flat_system_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12419,6 +13187,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_store( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12430,6 +13202,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_store( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12452,6 +13228,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12462,6 +13240,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12542,6 +13322,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_store( ; GFX7-LABEL: flat_system_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12554,6 +13337,10 @@ define amdgpu_kernel void @flat_system_one_as_release_store( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12567,6 +13354,10 @@ define amdgpu_kernel void @flat_system_one_as_release_store( ; ; GFX10-CU-LABEL: flat_system_one_as_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12592,6 +13383,8 @@ define amdgpu_kernel void @flat_system_one_as_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12604,6 +13397,8 @@ define amdgpu_kernel void @flat_system_one_as_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12704,6 +13499,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_store( ; GFX7-LABEL: flat_system_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12716,6 +13514,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12729,6 +13531,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_store( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12754,6 +13560,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12766,6 +13574,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12866,6 +13676,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_atomicrmw( ; GFX7-LABEL: flat_system_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12877,6 +13690,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12888,6 +13705,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12910,6 +13731,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12920,6 +13743,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13000,6 +13825,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_atomicrmw( ; GFX7-LABEL: flat_system_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13013,6 +13841,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13027,6 +13859,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13053,6 +13889,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13066,6 +13904,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13163,6 +14003,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_atomicrmw( ; GFX7-LABEL: flat_system_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13175,6 +14018,10 @@ define amdgpu_kernel void @flat_system_one_as_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13188,6 +14035,10 @@ define amdgpu_kernel void @flat_system_one_as_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13213,6 +14064,8 @@ define amdgpu_kernel void @flat_system_one_as_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13225,6 +14078,8 @@ define amdgpu_kernel void @flat_system_one_as_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13325,6 +14180,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_atomicrmw( ; GFX7-LABEL: flat_system_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13339,6 +14197,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13355,6 +14217,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13384,6 +14250,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13399,6 +14267,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13516,6 +14386,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_atomicrmw( ; GFX7-LABEL: flat_system_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13530,6 +14403,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13546,6 +14423,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13575,6 +14456,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13590,6 +14473,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13707,6 +14592,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_system_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13724,6 +14612,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13742,6 +14634,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13776,6 +14672,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13792,6 +14690,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -13913,6 +14813,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_system_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13931,6 +14834,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -13951,6 +14858,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -13988,6 +14899,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -14006,6 +14919,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -14151,6 +15066,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_system_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -14169,6 +15087,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -14189,6 +15111,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -14226,6 +15152,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -14244,6 +15172,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -14389,6 +15319,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14414,6 +15347,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14439,6 +15376,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14489,6 +15430,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14503,6 +15446,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14612,6 +15557,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14639,6 +15587,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14667,6 +15619,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14721,6 +15677,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14738,6 +15696,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14864,6 +15824,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14890,6 +15853,10 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14917,6 +15884,10 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14970,6 +15941,8 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14986,6 +15959,8 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15115,6 +16090,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15143,6 +16121,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15173,6 +16155,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15230,6 +16216,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15249,6 +16237,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15395,6 +16385,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_system_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15423,6 +16416,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15453,6 +16450,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15510,6 +16511,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15529,6 +16532,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15675,6 +16680,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_system_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15702,6 +16710,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15730,6 +16742,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15784,6 +16800,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15801,6 +16819,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15927,6 +16947,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15954,6 +16977,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15982,6 +17009,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16036,6 +17067,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16053,6 +17086,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16179,6 +17214,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_acquire_cmpxchg( ; GFX7-LABEL: flat_system_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16207,6 +17245,10 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16237,6 +17279,10 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16294,6 +17340,8 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16313,6 +17361,8 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16459,6 +17509,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16487,6 +17540,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16517,6 +17574,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16574,6 +17635,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16593,6 +17656,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16739,6 +17804,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_system_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16767,6 +17835,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16797,6 +17869,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -16854,6 +17930,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16873,6 +17951,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17019,6 +18099,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17047,6 +18130,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17077,6 +18164,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17134,6 +18225,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17153,6 +18246,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17299,6 +18394,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17327,6 +18425,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17357,6 +18459,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17414,6 +18520,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17433,6 +18541,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17579,6 +18689,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17607,6 +18720,10 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17637,6 +18754,10 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17694,6 +18815,8 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17713,6 +18836,8 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17859,6 +18984,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17887,6 +19015,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17917,6 +19049,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -17974,6 +19110,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17993,6 +19131,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18139,6 +19279,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_system_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -18167,6 +19310,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -18197,6 +19344,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -18254,6 +19405,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18273,6 +19426,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18419,6 +19574,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18448,6 +19606,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18477,6 +19639,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18535,6 +19701,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18552,6 +19720,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18688,6 +19858,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18719,6 +19892,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18751,6 +19928,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18813,6 +19994,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18833,6 +20016,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18984,6 +20169,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19014,6 +20202,10 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19045,6 +20237,10 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19106,6 +20302,8 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19125,6 +20323,8 @@ define amdgpu_kernel void @flat_system_one_as_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19281,6 +20481,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19313,6 +20516,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19347,6 +20554,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19412,6 +20623,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19434,6 +20647,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19609,6 +20824,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19641,6 +20859,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19675,6 +20897,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19740,6 +20966,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19762,6 +20990,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19937,6 +21167,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19968,6 +21201,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20000,6 +21237,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20062,6 +21303,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20082,6 +21325,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20237,6 +21482,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20268,6 +21516,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20300,6 +21552,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20362,6 +21618,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20382,6 +21640,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20533,6 +21793,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20565,6 +21828,10 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20599,6 +21866,10 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20664,6 +21935,8 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20686,6 +21959,8 @@ define amdgpu_kernel void @flat_system_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20861,6 +22136,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20893,6 +22171,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20927,6 +22209,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20992,6 +22278,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21014,6 +22302,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21189,6 +22479,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21221,6 +22514,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21255,6 +22552,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21320,6 +22621,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21342,6 +22645,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21517,6 +22822,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21549,6 +22857,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21583,6 +22895,10 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21648,6 +22964,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21670,6 +22988,8 @@ define amdgpu_kernel void @flat_system_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21845,6 +23165,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21877,6 +23200,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21911,6 +23238,10 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -21976,6 +23307,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -21998,6 +23331,8 @@ define amdgpu_kernel void @flat_system_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22169,6 +23504,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -22201,6 +23539,10 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22235,6 +23577,10 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22300,6 +23646,8 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22322,6 +23670,8 @@ define amdgpu_kernel void @flat_system_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22497,6 +23847,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -22529,6 +23882,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22563,6 +23920,10 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22628,6 +23989,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22650,6 +24013,8 @@ define amdgpu_kernel void @flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22825,6 +24190,9 @@ entry: define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -22857,6 +24225,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22891,6 +24263,10 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -22956,6 +24332,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -22978,6 +24356,8 @@ define amdgpu_kernel void @flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_system_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-volatile.ll index e1f82a70b4c0a..a88e0e217fdb4 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-volatile.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-volatile.ll @@ -11,6 +11,9 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; GFX7-LABEL: flat_nontemporal_load_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -26,6 +29,10 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; ; GFX10-WGP-LABEL: flat_nontemporal_load_0: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -41,6 +48,10 @@ define amdgpu_kernel void @flat_nontemporal_load_0( ; ; GFX10-CU-LABEL: flat_nontemporal_load_0: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -142,6 +153,9 @@ entry: define amdgpu_kernel void @flat_nontemporal_load_1( ; GFX7-LABEL: flat_nontemporal_load_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[4:5], 0x2 @@ -172,6 +186,10 @@ define amdgpu_kernel void @flat_nontemporal_load_1( ; ; GFX10-WGP-LABEL: flat_nontemporal_load_1: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_nop 0 @@ -202,6 +220,10 @@ define amdgpu_kernel void @flat_nontemporal_load_1( ; ; GFX10-CU-LABEL: flat_nontemporal_load_1: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_nop 0 @@ -405,6 +427,9 @@ entry: define amdgpu_kernel void @flat_nontemporal_store_0( ; GFX7-LABEL: flat_nontemporal_store_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -420,6 +445,10 @@ define amdgpu_kernel void @flat_nontemporal_store_0( ; ; GFX10-WGP-LABEL: flat_nontemporal_store_0: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -435,6 +464,10 @@ define amdgpu_kernel void @flat_nontemporal_store_0( ; ; GFX10-CU-LABEL: flat_nontemporal_store_0: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -540,6 +573,9 @@ entry: define amdgpu_kernel void @flat_nontemporal_store_1( ; GFX7-LABEL: flat_nontemporal_store_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -569,6 +605,10 @@ define amdgpu_kernel void @flat_nontemporal_store_1( ; ; GFX10-WGP-LABEL: flat_nontemporal_store_1: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -597,6 +637,10 @@ define amdgpu_kernel void @flat_nontemporal_store_1( ; ; GFX10-CU-LABEL: flat_nontemporal_store_1: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -799,6 +843,9 @@ entry: define amdgpu_kernel void @flat_volatile_workgroup_acquire_load( ; GFX7-LABEL: flat_volatile_workgroup_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -814,6 +861,10 @@ define amdgpu_kernel void @flat_volatile_workgroup_acquire_load( ; ; GFX10-WGP-LABEL: flat_volatile_workgroup_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -829,6 +880,10 @@ define amdgpu_kernel void @flat_volatile_workgroup_acquire_load( ; ; GFX10-CU-LABEL: flat_volatile_workgroup_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -926,6 +981,9 @@ entry: define amdgpu_kernel void @flat_volatile_workgroup_release_store( ; GFX7-LABEL: flat_volatile_workgroup_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -938,6 +996,10 @@ define amdgpu_kernel void @flat_volatile_workgroup_release_store( ; ; GFX10-WGP-LABEL: flat_volatile_workgroup_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -951,6 +1013,10 @@ define amdgpu_kernel void @flat_volatile_workgroup_release_store( ; ; GFX10-CU-LABEL: flat_volatile_workgroup_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-wavefront.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-wavefront.ll index 588f06f1be054..7c637a20ab47b 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-wavefront.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-wavefront.ll @@ -15,6 +15,9 @@ define amdgpu_kernel void @flat_wavefront_unordered_load( ; GFX7-LABEL: flat_wavefront_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -29,6 +32,10 @@ define amdgpu_kernel void @flat_wavefront_unordered_load( ; ; GFX10-WGP-LABEL: flat_wavefront_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -43,6 +50,10 @@ define amdgpu_kernel void @flat_wavefront_unordered_load( ; ; GFX10-CU-LABEL: flat_wavefront_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -71,6 +82,8 @@ define amdgpu_kernel void @flat_wavefront_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -83,6 +96,8 @@ define amdgpu_kernel void @flat_wavefront_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -182,6 +197,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_load( ; GFX7-LABEL: flat_wavefront_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -196,6 +214,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_load( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -210,6 +232,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_load( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -238,6 +264,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -250,6 +278,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -349,6 +379,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_load( ; GFX7-LABEL: flat_wavefront_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -363,6 +396,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_load( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -377,6 +414,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_load( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -405,6 +446,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -417,6 +460,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -516,6 +561,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_load( ; GFX7-LABEL: flat_wavefront_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -530,6 +578,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -544,6 +596,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_load( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -572,6 +628,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -584,6 +642,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -683,6 +743,9 @@ entry: define amdgpu_kernel void @flat_wavefront_unordered_store( ; GFX7-LABEL: flat_wavefront_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -694,6 +757,10 @@ define amdgpu_kernel void @flat_wavefront_unordered_store( ; ; GFX10-WGP-LABEL: flat_wavefront_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -705,6 +772,10 @@ define amdgpu_kernel void @flat_wavefront_unordered_store( ; ; GFX10-CU-LABEL: flat_wavefront_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -727,6 +798,8 @@ define amdgpu_kernel void @flat_wavefront_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -737,6 +810,8 @@ define amdgpu_kernel void @flat_wavefront_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -817,6 +892,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_store( ; GFX7-LABEL: flat_wavefront_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -828,6 +906,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_store( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -839,6 +921,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_store( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -861,6 +947,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -871,6 +959,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -951,6 +1041,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_store( ; GFX7-LABEL: flat_wavefront_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -962,6 +1055,10 @@ define amdgpu_kernel void @flat_wavefront_release_store( ; ; GFX10-WGP-LABEL: flat_wavefront_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -973,6 +1070,10 @@ define amdgpu_kernel void @flat_wavefront_release_store( ; ; GFX10-CU-LABEL: flat_wavefront_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -995,6 +1096,8 @@ define amdgpu_kernel void @flat_wavefront_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1005,6 +1108,8 @@ define amdgpu_kernel void @flat_wavefront_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1085,6 +1190,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_store( ; GFX7-LABEL: flat_wavefront_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1096,6 +1204,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1107,6 +1219,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_store( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1129,6 +1245,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1139,6 +1257,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1219,6 +1339,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_atomicrmw( ; GFX7-LABEL: flat_wavefront_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1230,6 +1353,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1241,6 +1368,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1263,6 +1394,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1273,6 +1406,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1353,6 +1488,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_atomicrmw( ; GFX7-LABEL: flat_wavefront_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1364,6 +1502,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1375,6 +1517,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1397,6 +1543,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1407,6 +1555,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1487,6 +1637,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_atomicrmw( ; GFX7-LABEL: flat_wavefront_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1498,6 +1651,10 @@ define amdgpu_kernel void @flat_wavefront_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1509,6 +1666,10 @@ define amdgpu_kernel void @flat_wavefront_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1531,6 +1692,8 @@ define amdgpu_kernel void @flat_wavefront_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1541,6 +1704,8 @@ define amdgpu_kernel void @flat_wavefront_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1621,6 +1786,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_atomicrmw( ; GFX7-LABEL: flat_wavefront_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1632,6 +1800,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1643,6 +1815,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1665,6 +1841,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1675,6 +1853,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1755,6 +1935,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_atomicrmw( ; GFX7-LABEL: flat_wavefront_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1766,6 +1949,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1777,6 +1964,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1799,6 +1990,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1809,6 +2002,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1889,6 +2084,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_wavefront_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1904,6 +2102,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1919,6 +2121,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1949,6 +2155,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1962,6 +2170,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2068,6 +2278,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_wavefront_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2083,6 +2296,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2098,6 +2315,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2128,6 +2349,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2141,6 +2364,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2247,6 +2472,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_wavefront_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2262,6 +2490,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2277,6 +2509,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2307,6 +2543,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2320,6 +2558,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2426,6 +2666,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2451,6 +2694,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2476,6 +2723,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2526,6 +2777,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2540,6 +2793,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2649,6 +2904,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2674,6 +2932,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2699,6 +2961,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2749,6 +3015,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2763,6 +3031,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2872,6 +3142,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2897,6 +3170,10 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2922,6 +3199,10 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2972,6 +3253,8 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2986,6 +3269,8 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3095,6 +3380,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3120,6 +3408,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3145,6 +3437,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3195,6 +3491,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3209,6 +3507,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3318,6 +3618,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3343,6 +3646,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3368,6 +3675,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3418,6 +3729,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3432,6 +3745,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3541,6 +3856,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3566,6 +3884,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3591,6 +3913,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3641,6 +3967,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3655,6 +3983,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3764,6 +4094,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3789,6 +4122,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3814,6 +4151,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3864,6 +4205,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3878,6 +4221,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3987,6 +4332,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4012,6 +4360,10 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4037,6 +4389,10 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4087,6 +4443,8 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4101,6 +4459,8 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4210,6 +4570,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4235,6 +4598,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4260,6 +4627,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4310,6 +4681,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4324,6 +4697,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4433,6 +4808,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4458,6 +4836,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4483,6 +4865,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4533,6 +4919,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4547,6 +4935,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4656,6 +5046,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4681,6 +5074,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4706,6 +5103,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4756,6 +5157,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4770,6 +5173,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4879,6 +5284,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4904,6 +5312,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4929,6 +5341,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4979,6 +5395,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4993,6 +5411,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5102,6 +5522,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5127,6 +5550,10 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5152,6 +5579,10 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5202,6 +5633,8 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5216,6 +5649,8 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5325,6 +5760,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5350,6 +5788,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5375,6 +5817,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5425,6 +5871,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5439,6 +5887,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5548,6 +5998,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5573,6 +6026,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5598,6 +6055,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5648,6 +6109,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5662,6 +6125,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5771,6 +6236,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -5800,6 +6268,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5829,6 +6301,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5887,6 +6363,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5904,6 +6382,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6040,6 +6520,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6069,6 +6552,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6098,6 +6585,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6156,6 +6647,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6173,6 +6666,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6309,6 +6804,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6338,6 +6836,10 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6367,6 +6869,10 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6425,6 +6931,8 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6442,6 +6950,8 @@ define amdgpu_kernel void @flat_wavefront_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6578,6 +7088,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6607,6 +7120,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6636,6 +7153,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6694,6 +7215,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6711,6 +7234,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6847,6 +7372,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6876,6 +7404,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6905,6 +7437,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6963,6 +7499,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6980,6 +7518,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7116,6 +7656,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7145,6 +7688,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7174,6 +7721,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7232,6 +7783,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7249,6 +7802,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7385,6 +7940,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7414,6 +7972,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7443,6 +8005,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7501,6 +8067,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7518,6 +8086,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7654,6 +8224,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7683,6 +8256,10 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7712,6 +8289,10 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7770,6 +8351,8 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7787,6 +8370,8 @@ define amdgpu_kernel void @flat_wavefront_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7923,6 +8508,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7952,6 +8540,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7981,6 +8573,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8039,6 +8635,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8056,6 +8654,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8192,6 +8792,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8221,6 +8824,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8250,6 +8857,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8308,6 +8919,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8325,6 +8938,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8461,6 +9076,9 @@ entry: define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8490,6 +9108,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8519,6 +9141,10 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8577,6 +9203,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8594,6 +9222,8 @@ define amdgpu_kernel void @flat_wavefront_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8730,6 +9360,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8759,6 +9392,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8788,6 +9425,10 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8846,6 +9487,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8863,6 +9506,8 @@ define amdgpu_kernel void @flat_wavefront_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8999,6 +9644,9 @@ entry: define amdgpu_kernel void @flat_wavefront_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9028,6 +9676,10 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9057,6 +9709,10 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9115,6 +9771,8 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9132,6 +9790,8 @@ define amdgpu_kernel void @flat_wavefront_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9268,6 +9928,9 @@ entry: define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9297,6 +9960,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9326,6 +9993,10 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9384,6 +10055,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9401,6 +10074,8 @@ define amdgpu_kernel void @flat_wavefront_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9537,6 +10212,9 @@ entry: define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9566,6 +10244,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9595,6 +10277,10 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9653,6 +10339,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9670,6 +10358,8 @@ define amdgpu_kernel void @flat_wavefront_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9806,6 +10496,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_unordered_load( ; GFX7-LABEL: flat_wavefront_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9820,6 +10513,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_load( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -9834,6 +10531,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_load( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -9862,6 +10563,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -9874,6 +10577,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -9973,6 +10678,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_load( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9987,6 +10695,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_load( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10001,6 +10713,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_load( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10029,6 +10745,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10041,6 +10759,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10140,6 +10860,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_load( ; GFX7-LABEL: flat_wavefront_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10154,6 +10877,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_load( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10168,6 +10895,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_load( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10196,6 +10927,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10208,6 +10941,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10307,6 +11042,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_load( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10321,6 +11059,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10335,6 +11077,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_load( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10363,6 +11109,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10375,6 +11123,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10474,6 +11224,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_unordered_store( ; GFX7-LABEL: flat_wavefront_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10485,6 +11238,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_store( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10496,6 +11253,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_store( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10518,6 +11279,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10528,6 +11291,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10608,6 +11373,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_store( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10619,6 +11387,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_store( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10630,6 +11402,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_store( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10652,6 +11428,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10662,6 +11440,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10742,6 +11522,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_release_store( ; GFX7-LABEL: flat_wavefront_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10753,6 +11536,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_store( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10764,6 +11551,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_store( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10786,6 +11577,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10796,6 +11589,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10876,6 +11671,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_store( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10887,6 +11685,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10898,6 +11700,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_store( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10920,6 +11726,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10930,6 +11738,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11010,6 +11820,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11021,6 +11834,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11032,6 +11849,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11054,6 +11875,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11064,6 +11887,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11144,6 +11969,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11155,6 +11983,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11166,6 +11998,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11188,6 +12024,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11198,6 +12036,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11278,6 +12118,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_release_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11289,6 +12132,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11300,6 +12147,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11322,6 +12173,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11332,6 +12185,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11412,6 +12267,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11423,6 +12281,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11434,6 +12296,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11456,6 +12322,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11466,6 +12334,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11546,6 +12416,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11557,6 +12430,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11568,6 +12445,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11590,6 +12471,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11600,6 +12483,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11680,6 +12565,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11695,6 +12583,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11710,6 +12602,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11740,6 +12636,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11753,6 +12651,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11859,6 +12759,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11874,6 +12777,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11889,6 +12796,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11919,6 +12830,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11932,6 +12845,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12038,6 +12953,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12053,6 +12971,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12068,6 +12990,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12098,6 +13024,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12111,6 +13039,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12217,6 +13147,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12242,6 +13175,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12267,6 +13204,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12317,6 +13258,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12331,6 +13274,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12440,6 +13385,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12465,6 +13413,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12490,6 +13442,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12540,6 +13496,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12554,6 +13512,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12663,6 +13623,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12688,6 +13651,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12713,6 +13680,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12763,6 +13734,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12777,6 +13750,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12886,6 +13861,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12911,6 +13889,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12936,6 +13918,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12986,6 +13972,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13000,6 +13988,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13109,6 +14099,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13134,6 +14127,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13159,6 +14156,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13209,6 +14210,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13223,6 +14226,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13332,6 +14337,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13357,6 +14365,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13382,6 +14394,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13432,6 +14448,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13446,6 +14464,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13555,6 +14575,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13580,6 +14603,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13605,6 +14632,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13655,6 +14686,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13669,6 +14702,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13778,6 +14813,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13803,6 +14841,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13828,6 +14870,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13878,6 +14924,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13892,6 +14940,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14001,6 +15051,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14026,6 +15079,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14051,6 +15108,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14101,6 +15162,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14115,6 +15178,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14224,6 +15289,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14249,6 +15317,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14274,6 +15346,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14324,6 +15400,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14338,6 +15416,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14447,6 +15527,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14472,6 +15555,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14497,6 +15584,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14547,6 +15638,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14561,6 +15654,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14670,6 +15765,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14695,6 +15793,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14720,6 +15822,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14770,6 +15876,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14784,6 +15892,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14893,6 +16003,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14918,6 +16031,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14943,6 +16060,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14993,6 +16114,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15007,6 +16130,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15116,6 +16241,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15141,6 +16269,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15166,6 +16298,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15216,6 +16352,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15230,6 +16368,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15339,6 +16479,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15364,6 +16507,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15389,6 +16536,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15439,6 +16590,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15453,6 +16606,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15562,6 +16717,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15591,6 +16749,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15620,6 +16782,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15678,6 +16844,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15695,6 +16863,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15831,6 +17001,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15860,6 +17033,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15889,6 +17066,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -15947,6 +17128,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15964,6 +17147,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16100,6 +17285,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16129,6 +17317,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16158,6 +17350,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16216,6 +17412,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16233,6 +17431,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16369,6 +17569,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16398,6 +17601,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16427,6 +17634,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16485,6 +17696,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16502,6 +17715,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16638,6 +17853,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16667,6 +17885,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16696,6 +17918,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16754,6 +17980,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16771,6 +17999,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16907,6 +18137,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16936,6 +18169,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16965,6 +18202,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17023,6 +18264,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17040,6 +18283,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17176,6 +18421,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17205,6 +18453,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17234,6 +18486,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17292,6 +18548,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17309,6 +18567,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17445,6 +18705,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17474,6 +18737,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17503,6 +18770,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17561,6 +18832,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17578,6 +18851,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17714,6 +18989,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17743,6 +19021,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17772,6 +19054,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17830,6 +19116,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17847,6 +19135,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17983,6 +19273,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18012,6 +19305,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18041,6 +19338,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18099,6 +19400,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18116,6 +19419,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18252,6 +19557,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18281,6 +19589,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18310,6 +19622,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18368,6 +19684,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18385,6 +19703,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18521,6 +19841,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18550,6 +19873,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18579,6 +19906,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18637,6 +19968,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18654,6 +19987,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18790,6 +20125,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18819,6 +20157,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18848,6 +20190,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18906,6 +20252,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18923,6 +20271,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_acq_relc_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19059,6 +20409,9 @@ entry: define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19088,6 +20441,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19117,6 +20474,10 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19175,6 +20536,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19192,6 +20555,8 @@ define amdgpu_kernel void @flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-workgroup.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-workgroup.ll index ee7d79a8a8cbb..0fd4aa4a7a93f 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-workgroup.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-flat-workgroup.ll @@ -15,6 +15,9 @@ define amdgpu_kernel void @flat_workgroup_unordered_load( ; GFX7-LABEL: flat_workgroup_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -29,6 +32,10 @@ define amdgpu_kernel void @flat_workgroup_unordered_load( ; ; GFX10-WGP-LABEL: flat_workgroup_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -43,6 +50,10 @@ define amdgpu_kernel void @flat_workgroup_unordered_load( ; ; GFX10-CU-LABEL: flat_workgroup_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -71,6 +82,8 @@ define amdgpu_kernel void @flat_workgroup_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -83,6 +96,8 @@ define amdgpu_kernel void @flat_workgroup_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -182,6 +197,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_load( ; GFX7-LABEL: flat_workgroup_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -196,6 +214,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_load( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -210,6 +232,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_load( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -238,6 +264,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -250,6 +278,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -349,6 +379,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_load( ; GFX7-LABEL: flat_workgroup_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -364,6 +397,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_load( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -379,6 +416,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_load( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -409,6 +450,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -422,6 +465,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -528,6 +573,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_load( ; GFX7-LABEL: flat_workgroup_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -544,6 +592,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -561,6 +613,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_load( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -593,6 +649,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -607,6 +665,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -726,6 +786,9 @@ entry: define amdgpu_kernel void @flat_workgroup_unordered_store( ; GFX7-LABEL: flat_workgroup_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -737,6 +800,10 @@ define amdgpu_kernel void @flat_workgroup_unordered_store( ; ; GFX10-WGP-LABEL: flat_workgroup_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -748,6 +815,10 @@ define amdgpu_kernel void @flat_workgroup_unordered_store( ; ; GFX10-CU-LABEL: flat_workgroup_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -770,6 +841,8 @@ define amdgpu_kernel void @flat_workgroup_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -780,6 +853,8 @@ define amdgpu_kernel void @flat_workgroup_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -860,6 +935,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_store( ; GFX7-LABEL: flat_workgroup_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -871,6 +949,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_store( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -882,6 +964,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_store( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -904,6 +990,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -914,6 +1002,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -994,6 +1084,9 @@ entry: define amdgpu_kernel void @flat_workgroup_release_store( ; GFX7-LABEL: flat_workgroup_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1006,6 +1099,10 @@ define amdgpu_kernel void @flat_workgroup_release_store( ; ; GFX10-WGP-LABEL: flat_workgroup_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1019,6 +1116,10 @@ define amdgpu_kernel void @flat_workgroup_release_store( ; ; GFX10-CU-LABEL: flat_workgroup_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1043,6 +1144,8 @@ define amdgpu_kernel void @flat_workgroup_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1054,6 +1157,8 @@ define amdgpu_kernel void @flat_workgroup_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1145,6 +1250,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_store( ; GFX7-LABEL: flat_workgroup_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1157,6 +1265,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1170,6 +1282,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_store( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1194,6 +1310,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1205,6 +1323,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1296,6 +1416,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_atomicrmw( ; GFX7-LABEL: flat_workgroup_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1307,6 +1430,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1318,6 +1445,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1340,6 +1471,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1350,6 +1483,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1430,6 +1565,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_atomicrmw( ; GFX7-LABEL: flat_workgroup_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1442,6 +1580,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1456,6 +1598,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1480,6 +1626,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1491,6 +1639,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1583,6 +1733,9 @@ entry: define amdgpu_kernel void @flat_workgroup_release_atomicrmw( ; GFX7-LABEL: flat_workgroup_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1595,6 +1748,10 @@ define amdgpu_kernel void @flat_workgroup_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1608,6 +1765,10 @@ define amdgpu_kernel void @flat_workgroup_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1632,6 +1793,8 @@ define amdgpu_kernel void @flat_workgroup_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1643,6 +1806,8 @@ define amdgpu_kernel void @flat_workgroup_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1734,6 +1899,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acq_rel_atomicrmw( ; GFX7-LABEL: flat_workgroup_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1747,6 +1915,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1763,6 +1935,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1789,6 +1965,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1801,6 +1979,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1904,6 +2084,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_atomicrmw( ; GFX7-LABEL: flat_workgroup_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1917,6 +2100,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -1933,6 +2120,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -1959,6 +2150,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -1971,6 +2164,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2074,6 +2269,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_workgroup_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2090,6 +2288,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2106,6 +2308,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2138,6 +2344,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2152,6 +2360,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2265,6 +2475,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_workgroup_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2282,6 +2495,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2300,6 +2517,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2334,6 +2555,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2349,6 +2572,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2475,6 +2700,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_workgroup_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2492,6 +2720,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -2510,6 +2742,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -2544,6 +2780,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2559,6 +2797,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -2685,6 +2925,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2710,6 +2953,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2735,6 +2982,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2785,6 +3036,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2799,6 +3052,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -2908,6 +3163,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2934,6 +3192,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -2962,6 +3224,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3014,6 +3280,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3029,6 +3297,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3150,6 +3420,9 @@ entry: define amdgpu_kernel void @flat_workgroup_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3176,6 +3449,10 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3203,6 +3480,10 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3255,6 +3536,8 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3270,6 +3553,8 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3390,6 +3675,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3417,6 +3705,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3447,6 +3739,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3501,6 +3797,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3517,6 +3815,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3649,6 +3949,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3676,6 +3979,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3706,6 +4013,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3760,6 +4071,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3776,6 +4089,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -3908,6 +4223,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3934,6 +4252,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -3962,6 +4284,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4014,6 +4340,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4029,6 +4357,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4150,6 +4480,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4176,6 +4509,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4204,6 +4541,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4256,6 +4597,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4271,6 +4614,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4392,6 +4737,9 @@ entry: define amdgpu_kernel void @flat_workgroup_release_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4419,6 +4767,10 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4449,6 +4801,10 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4503,6 +4859,8 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4519,6 +4877,8 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4651,6 +5011,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4678,6 +5041,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4708,6 +5075,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4762,6 +5133,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4778,6 +5151,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -4910,6 +5285,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4937,6 +5315,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -4967,6 +5349,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5021,6 +5407,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5037,6 +5425,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5169,6 +5559,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_workgroup_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5196,6 +5589,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5226,6 +5623,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -5280,6 +5681,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5296,6 +5699,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5428,6 +5833,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -5457,6 +5865,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5486,6 +5898,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5544,6 +5960,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5561,6 +5979,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5697,6 +6117,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -5727,6 +6150,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5757,6 +6184,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -5817,6 +6248,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5835,6 +6268,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -5978,6 +6413,9 @@ entry: define amdgpu_kernel void @flat_workgroup_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6008,6 +6446,10 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6039,6 +6481,10 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6099,6 +6545,8 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6117,6 +6565,8 @@ define amdgpu_kernel void @flat_workgroup_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6264,6 +6714,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6295,6 +6748,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6327,6 +6784,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6389,6 +6850,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6408,6 +6871,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6564,6 +7029,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6595,6 +7063,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6627,6 +7099,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6689,6 +7165,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6708,6 +7186,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -6864,6 +7344,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6894,6 +7377,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6924,6 +7411,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -6984,6 +7475,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7002,6 +7495,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7147,6 +7642,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7177,6 +7675,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7207,6 +7709,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7267,6 +7773,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7285,6 +7793,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7428,6 +7938,9 @@ entry: define amdgpu_kernel void @flat_workgroup_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7459,6 +7972,10 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7491,6 +8008,10 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7553,6 +8074,8 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7572,6 +8095,8 @@ define amdgpu_kernel void @flat_workgroup_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7728,6 +8253,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7759,6 +8287,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7791,6 +8323,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -7853,6 +8389,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -7872,6 +8410,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8028,6 +8568,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8059,6 +8602,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8091,6 +8638,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8153,6 +8704,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8172,6 +8725,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8328,6 +8883,9 @@ entry: define amdgpu_kernel void @flat_workgroup_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8359,6 +8917,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8391,6 +8953,10 @@ define amdgpu_kernel void @flat_workgroup_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8453,6 +9019,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8472,6 +9040,8 @@ define amdgpu_kernel void @flat_workgroup_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8628,6 +9198,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8659,6 +9232,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8691,6 +9268,10 @@ define amdgpu_kernel void @flat_workgroup_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8753,6 +9334,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8772,6 +9355,8 @@ define amdgpu_kernel void @flat_workgroup_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -8926,6 +9511,9 @@ entry: define amdgpu_kernel void @flat_workgroup_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8957,6 +9545,10 @@ define amdgpu_kernel void @flat_workgroup_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -8989,6 +9581,10 @@ define amdgpu_kernel void @flat_workgroup_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9051,6 +9647,8 @@ define amdgpu_kernel void @flat_workgroup_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9070,6 +9668,8 @@ define amdgpu_kernel void @flat_workgroup_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9226,6 +9826,9 @@ entry: define amdgpu_kernel void @flat_workgroup_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9257,6 +9860,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9289,6 +9896,10 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9351,6 +9962,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9370,6 +9983,8 @@ define amdgpu_kernel void @flat_workgroup_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9526,6 +10141,9 @@ entry: define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9557,6 +10175,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9589,6 +10211,10 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -9651,6 +10277,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9670,6 +10298,8 @@ define amdgpu_kernel void @flat_workgroup_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -9826,6 +10456,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_unordered_load( ; GFX7-LABEL: flat_workgroup_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9840,6 +10473,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_load( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_unordered_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -9854,6 +10491,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_load( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_unordered_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -9882,6 +10523,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_unordered_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -9894,6 +10537,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_unordered_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -9993,6 +10638,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_load( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10007,6 +10655,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_load( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10021,6 +10673,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_load( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10049,6 +10705,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10061,6 +10719,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10160,6 +10820,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_load( ; GFX7-LABEL: flat_workgroup_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10174,6 +10837,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_load( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10190,6 +10857,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_load( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10218,6 +10889,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10230,6 +10903,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10335,6 +11010,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_load( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10349,6 +11027,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_load( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_load: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10367,6 +11049,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_load( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_load: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10395,6 +11081,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_load( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10407,6 +11095,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_load( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10522,6 +11212,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_unordered_store( ; GFX7-LABEL: flat_workgroup_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10533,6 +11226,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_store( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_unordered_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10544,6 +11241,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_store( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_unordered_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10566,6 +11267,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_unordered_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10576,6 +11279,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_unordered_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_unordered_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10656,6 +11361,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_store( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10667,6 +11375,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_store( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10678,6 +11390,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_store( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10700,6 +11416,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10710,6 +11428,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10790,6 +11510,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_store( ; GFX7-LABEL: flat_workgroup_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10801,6 +11524,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_store( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10814,6 +11541,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_store( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10836,6 +11567,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10846,6 +11579,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10934,6 +11669,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_store( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10945,6 +11683,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_store( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_store: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -10958,6 +11700,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_store( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_store: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -10980,6 +11726,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_store( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_store: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -10990,6 +11738,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_store( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_store: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11078,6 +11828,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11089,6 +11842,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11100,6 +11857,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11122,6 +11883,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11132,6 +11895,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11212,6 +11977,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11223,6 +11991,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11236,6 +12008,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11258,6 +12034,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11268,6 +12046,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11356,6 +12136,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11367,6 +12150,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11380,6 +12167,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11402,6 +12193,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11412,6 +12205,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11500,6 +12295,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11511,6 +12309,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11526,6 +12328,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11548,6 +12354,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11558,6 +12366,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11654,6 +12464,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11665,6 +12478,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11680,6 +12497,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11702,6 +12523,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11712,6 +12535,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11808,6 +12633,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_ret_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11823,6 +12651,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -11840,6 +12672,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -11870,6 +12706,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11883,6 +12721,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -11995,6 +12835,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_ret_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12010,6 +12853,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12029,6 +12876,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12059,6 +12910,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12072,6 +12925,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12194,6 +13049,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_ret_atomicrmw( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12209,6 +13067,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_ret_atomicrmw: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-WGP-NEXT: s_waitcnt lgkmcnt(0) @@ -12228,6 +13090,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_ret_atomicrmw( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_ret_atomicrmw: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX10-CU-NEXT: s_waitcnt lgkmcnt(0) @@ -12258,6 +13124,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_ret_atomicrmw: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12271,6 +13139,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_ret_atomicrmw( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_ret_atomicrmw: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_waitcnt lgkmcnt(0) @@ -12393,6 +13263,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12418,6 +13291,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12443,6 +13320,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12493,6 +13374,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12507,6 +13390,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12616,6 +13501,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12641,6 +13529,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12668,6 +13560,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12718,6 +13614,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12732,6 +13630,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12849,6 +13749,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12874,6 +13777,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12901,6 +13808,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -12951,6 +13862,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -12965,6 +13878,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13082,6 +13997,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13107,6 +14025,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13136,6 +14058,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13186,6 +14112,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13200,6 +14128,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13325,6 +14255,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13350,6 +14283,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13379,6 +14316,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_monotonic_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13429,6 +14370,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13443,6 +14386,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_monotonic_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13568,6 +14513,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13593,6 +14541,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13620,6 +14572,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13670,6 +14626,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13684,6 +14642,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13801,6 +14761,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13826,6 +14789,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13853,6 +14820,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -13903,6 +14874,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -13917,6 +14890,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14034,6 +15009,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14059,6 +15037,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14088,6 +15070,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14138,6 +15124,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14152,6 +15140,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14277,6 +15267,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14302,6 +15295,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14331,6 +15328,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14381,6 +15382,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14395,6 +15398,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14520,6 +15525,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14545,6 +15553,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_acquire_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14574,6 +15586,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_acquire_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14624,6 +15640,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14638,6 +15656,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_acquire_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14763,6 +15783,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14788,6 +15811,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14817,6 +15844,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -14867,6 +15898,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -14881,6 +15914,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15006,6 +16041,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15031,6 +16069,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15060,6 +16102,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15110,6 +16156,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15124,6 +16172,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15249,6 +16299,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15274,6 +16327,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15303,6 +16360,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15353,6 +16414,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15367,6 +16430,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15492,6 +16557,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15517,6 +16585,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15546,6 +16618,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15596,6 +16672,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15610,6 +16688,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15735,6 +16815,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15760,6 +16843,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-WGP-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15789,6 +16876,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX10-CU-NEXT: s_load_dword s7, s[4:5], 0x8 @@ -15839,6 +16930,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15853,6 +16946,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -15978,6 +17073,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16007,6 +17105,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16036,6 +17138,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16094,6 +17200,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16111,6 +17219,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonicmonotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16247,6 +17357,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16276,6 +17389,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16307,6 +17424,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16365,6 +17486,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16382,6 +17505,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16524,6 +17649,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16553,6 +17681,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16584,6 +17716,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16642,6 +17778,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16659,6 +17797,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16803,6 +17943,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16832,6 +17975,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16865,6 +18012,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -16923,6 +18074,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -16940,6 +18093,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17092,6 +18247,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17121,6 +18279,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17154,6 +18316,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17212,6 +18378,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17229,6 +18397,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17381,6 +18551,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17410,6 +18583,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17441,6 +18618,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17499,6 +18680,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17516,6 +18699,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17660,6 +18845,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17689,6 +18877,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17720,6 +18912,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17778,6 +18974,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17795,6 +18993,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -17937,6 +19137,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17966,6 +19169,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -17999,6 +19206,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18057,6 +19268,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18074,6 +19287,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18226,6 +19441,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18255,6 +19473,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18288,6 +19510,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18346,6 +19572,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18363,6 +19591,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18515,6 +19745,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18544,6 +19777,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18577,6 +19814,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18635,6 +19876,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18652,6 +19895,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18804,6 +20049,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18833,6 +20081,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18866,6 +20118,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -18924,6 +20180,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -18941,6 +20199,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19093,6 +20353,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19122,6 +20385,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19155,6 +20422,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19213,6 +20484,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19230,6 +20503,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19380,6 +20655,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19409,6 +20687,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19442,6 +20724,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_release_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19500,6 +20786,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19517,6 +20805,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_release_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19669,6 +20959,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19698,6 +20991,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19731,6 +21028,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -19789,6 +21090,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19806,6 +21109,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -19958,6 +21263,9 @@ entry: define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg( ; GFX7-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19987,6 +21295,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-WGP-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-WGP: ; %bb.0: ; %entry +; GFX10-WGP-NEXT: s_add_u32 s12, s12, s17 +; GFX10-WGP-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-WGP-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-WGP-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-WGP-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20020,6 +21332,10 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX10-CU-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX10-CU: ; %bb.0: ; %entry +; GFX10-CU-NEXT: s_add_u32 s12, s12, s17 +; GFX10-CU-NEXT: s_addc_u32 s13, s13, 0 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12 +; GFX10-CU-NEXT: s_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13 ; GFX10-CU-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX10-CU-NEXT: s_load_dword s9, s[6:7], 0x8 @@ -20078,6 +21394,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-NOTTGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc @@ -20095,6 +21413,8 @@ define amdgpu_kernel void @flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX90A-TGSPLIT-LABEL: flat_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry +; GFX90A-TGSPLIT-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-TGSPLIT-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x8 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0xc diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-agent.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-agent.ll index b9487f8e14c2b..8b600c835a160 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-agent.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-agent.ll @@ -41,6 +41,9 @@ define amdgpu_kernel void @global_agent_unordered_load( ; ; GFX7-LABEL: global_agent_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -222,6 +225,9 @@ define amdgpu_kernel void @global_agent_monotonic_load( ; ; GFX7-LABEL: global_agent_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -404,6 +410,9 @@ define amdgpu_kernel void @global_agent_acquire_load( ; ; GFX7-LABEL: global_agent_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -602,6 +611,9 @@ define amdgpu_kernel void @global_agent_seq_cst_load( ; ; GFX7-LABEL: global_agent_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -809,6 +821,9 @@ define amdgpu_kernel void @global_agent_unordered_store( ; ; GFX7-LABEL: global_agent_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -962,6 +977,9 @@ define amdgpu_kernel void @global_agent_monotonic_store( ; ; GFX7-LABEL: global_agent_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1116,6 +1134,9 @@ define amdgpu_kernel void @global_agent_release_store( ; ; GFX7-LABEL: global_agent_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1294,6 +1315,9 @@ define amdgpu_kernel void @global_agent_seq_cst_store( ; ; GFX7-LABEL: global_agent_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1470,6 +1494,9 @@ define amdgpu_kernel void @global_agent_monotonic_atomicrmw( ; ; GFX7-LABEL: global_agent_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1623,6 +1650,9 @@ define amdgpu_kernel void @global_agent_acquire_atomicrmw( ; ; GFX7-LABEL: global_agent_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1802,6 +1832,9 @@ define amdgpu_kernel void @global_agent_release_atomicrmw( ; ; GFX7-LABEL: global_agent_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1980,6 +2013,9 @@ define amdgpu_kernel void @global_agent_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_agent_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2185,6 +2221,9 @@ define amdgpu_kernel void @global_agent_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_agent_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2390,6 +2429,9 @@ define amdgpu_kernel void @global_agent_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_agent_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2587,6 +2629,9 @@ define amdgpu_kernel void @global_agent_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_agent_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2812,6 +2857,9 @@ define amdgpu_kernel void @global_agent_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_agent_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -3038,6 +3086,9 @@ define amdgpu_kernel void @global_agent_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3256,6 +3307,9 @@ define amdgpu_kernel void @global_agent_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3500,6 +3554,9 @@ define amdgpu_kernel void @global_agent_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3743,6 +3800,9 @@ define amdgpu_kernel void @global_agent_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4013,6 +4073,9 @@ define amdgpu_kernel void @global_agent_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4282,6 +4345,9 @@ define amdgpu_kernel void @global_agent_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4527,6 +4593,9 @@ define amdgpu_kernel void @global_agent_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4773,6 +4842,9 @@ define amdgpu_kernel void @global_agent_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5043,6 +5115,9 @@ define amdgpu_kernel void @global_agent_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5313,6 +5388,9 @@ define amdgpu_kernel void @global_agent_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5583,6 +5661,9 @@ define amdgpu_kernel void @global_agent_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5853,6 +5934,9 @@ define amdgpu_kernel void @global_agent_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6123,6 +6207,9 @@ define amdgpu_kernel void @global_agent_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6393,6 +6480,9 @@ define amdgpu_kernel void @global_agent_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6663,6 +6753,9 @@ define amdgpu_kernel void @global_agent_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6933,6 +7026,9 @@ define amdgpu_kernel void @global_agent_monotonic_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7182,6 +7278,9 @@ define amdgpu_kernel void @global_agent_acquire_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7447,6 +7546,9 @@ define amdgpu_kernel void @global_agent_release_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7721,6 +7823,9 @@ define amdgpu_kernel void @global_agent_acq_rel_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8015,6 +8120,9 @@ define amdgpu_kernel void @global_agent_seq_cst_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8308,6 +8416,9 @@ define amdgpu_kernel void @global_agent_monotonic_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8577,6 +8688,9 @@ define amdgpu_kernel void @global_agent_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8843,6 +8957,9 @@ define amdgpu_kernel void @global_agent_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9137,6 +9254,9 @@ define amdgpu_kernel void @global_agent_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9431,6 +9551,9 @@ define amdgpu_kernel void @global_agent_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9725,6 +9848,9 @@ define amdgpu_kernel void @global_agent_monotonic_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10019,6 +10145,9 @@ define amdgpu_kernel void @global_agent_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10309,6 +10438,9 @@ define amdgpu_kernel void @global_agent_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10603,6 +10735,9 @@ define amdgpu_kernel void @global_agent_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10897,6 +11032,9 @@ define amdgpu_kernel void @global_agent_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -11189,6 +11327,9 @@ define amdgpu_kernel void @global_agent_one_as_unordered_load( ; ; GFX7-LABEL: global_agent_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11370,6 +11511,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_load( ; ; GFX7-LABEL: global_agent_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11552,6 +11696,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_load( ; ; GFX7-LABEL: global_agent_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11750,6 +11897,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_load( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11957,6 +12107,9 @@ define amdgpu_kernel void @global_agent_one_as_unordered_store( ; ; GFX7-LABEL: global_agent_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12110,6 +12263,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_store( ; ; GFX7-LABEL: global_agent_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12264,6 +12420,9 @@ define amdgpu_kernel void @global_agent_one_as_release_store( ; ; GFX7-LABEL: global_agent_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12442,6 +12601,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_store( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12618,6 +12780,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12771,6 +12936,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12950,6 +13118,9 @@ define amdgpu_kernel void @global_agent_one_as_release_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13128,6 +13299,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13333,6 +13507,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13538,6 +13715,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13735,6 +13915,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13960,6 +14143,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -14186,6 +14372,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14404,6 +14593,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14648,6 +14840,9 @@ define amdgpu_kernel void @global_agent_one_as_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14891,6 +15086,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15161,6 +15359,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15430,6 +15631,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15675,6 +15879,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15921,6 +16128,9 @@ define amdgpu_kernel void @global_agent_one_as_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16191,6 +16401,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16461,6 +16674,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16731,6 +16947,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17001,6 +17220,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17271,6 +17493,9 @@ define amdgpu_kernel void @global_agent_one_as_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17541,6 +17766,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -17811,6 +18039,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -18081,6 +18312,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18330,6 +18564,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18596,6 +18833,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18890,6 +19130,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19183,6 +19426,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19452,6 +19698,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19718,6 +19967,9 @@ define amdgpu_kernel void @global_agent_one_as_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20012,6 +20264,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20306,6 +20561,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20600,6 +20858,9 @@ define amdgpu_kernel void @global_agent_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20894,6 +21155,9 @@ define amdgpu_kernel void @global_agent_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21184,6 +21448,9 @@ define amdgpu_kernel void @global_agent_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21478,6 +21745,9 @@ define amdgpu_kernel void @global_agent_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21772,6 +22042,9 @@ define amdgpu_kernel void @global_agent_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_agent_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-nontemporal.ll index a6bd1b678f95e..16e55058e4fc8 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-nontemporal.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-nontemporal.ll @@ -36,6 +36,9 @@ define amdgpu_kernel void @global_nontemporal_load_0( ; ; GFX7-LABEL: global_nontemporal_load_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -227,6 +230,9 @@ define amdgpu_kernel void @global_nontemporal_load_1( ; ; GFX7-LABEL: global_nontemporal_load_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[4:5], 0x2 @@ -474,6 +480,9 @@ define amdgpu_kernel void @global_nontemporal_store_0( ; ; GFX7-LABEL: global_nontemporal_store_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -658,6 +667,9 @@ define amdgpu_kernel void @global_nontemporal_store_1( ; ; GFX7-LABEL: global_nontemporal_store_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -891,6 +903,9 @@ define amdgpu_kernel void @global_nontemporal_volatile_load( ; ; GFX7-LABEL: global_nontemporal_volatile_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-singlethread.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-singlethread.ll index a5de6a92db1af..8042d38716107 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-singlethread.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-singlethread.ll @@ -41,6 +41,9 @@ define amdgpu_kernel void @global_singlethread_unordered_load( ; ; GFX7-LABEL: global_singlethread_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -222,6 +225,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_load( ; ; GFX7-LABEL: global_singlethread_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -403,6 +409,9 @@ define amdgpu_kernel void @global_singlethread_acquire_load( ; ; GFX7-LABEL: global_singlethread_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -584,6 +593,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_load( ; ; GFX7-LABEL: global_singlethread_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -758,6 +770,9 @@ define amdgpu_kernel void @global_singlethread_unordered_store( ; ; GFX7-LABEL: global_singlethread_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -911,6 +926,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_store( ; ; GFX7-LABEL: global_singlethread_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1064,6 +1082,9 @@ define amdgpu_kernel void @global_singlethread_release_store( ; ; GFX7-LABEL: global_singlethread_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1217,6 +1238,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_store( ; ; GFX7-LABEL: global_singlethread_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1369,6 +1393,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_atomicrmw( ; ; GFX7-LABEL: global_singlethread_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1520,6 +1547,9 @@ define amdgpu_kernel void @global_singlethread_acquire_atomicrmw( ; ; GFX7-LABEL: global_singlethread_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1671,6 +1701,9 @@ define amdgpu_kernel void @global_singlethread_release_atomicrmw( ; ; GFX7-LABEL: global_singlethread_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1822,6 +1855,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_singlethread_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1973,6 +2009,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_singlethread_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2126,6 +2165,9 @@ define amdgpu_kernel void @global_singlethread_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_singlethread_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2306,6 +2348,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_singlethread_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2486,6 +2531,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_singlethread_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2669,6 +2717,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2885,6 +2936,9 @@ define amdgpu_kernel void @global_singlethread_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3101,6 +3155,9 @@ define amdgpu_kernel void @global_singlethread_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3317,6 +3374,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3533,6 +3593,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3749,6 +3812,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3965,6 +4031,9 @@ define amdgpu_kernel void @global_singlethread_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4181,6 +4250,9 @@ define amdgpu_kernel void @global_singlethread_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4397,6 +4469,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4613,6 +4688,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4829,6 +4907,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5045,6 +5126,9 @@ define amdgpu_kernel void @global_singlethread_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5261,6 +5345,9 @@ define amdgpu_kernel void @global_singlethread_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5477,6 +5564,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5693,6 +5783,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5912,6 +6005,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6160,6 +6256,9 @@ define amdgpu_kernel void @global_singlethread_acquire_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6408,6 +6507,9 @@ define amdgpu_kernel void @global_singlethread_release_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6656,6 +6758,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6904,6 +7009,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7152,6 +7260,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7400,6 +7511,9 @@ define amdgpu_kernel void @global_singlethread_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7648,6 +7762,9 @@ define amdgpu_kernel void @global_singlethread_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7896,6 +8013,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8144,6 +8264,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8392,6 +8515,9 @@ define amdgpu_kernel void @global_singlethread_monotonic_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8640,6 +8766,9 @@ define amdgpu_kernel void @global_singlethread_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8888,6 +9017,9 @@ define amdgpu_kernel void @global_singlethread_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9136,6 +9268,9 @@ define amdgpu_kernel void @global_singlethread_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9384,6 +9519,9 @@ define amdgpu_kernel void @global_singlethread_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_singlethread_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9632,6 +9770,9 @@ define amdgpu_kernel void @global_singlethread_one_as_unordered_load( ; ; GFX7-LABEL: global_singlethread_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9813,6 +9954,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_load( ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9994,6 +10138,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_load( ; ; GFX7-LABEL: global_singlethread_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10175,6 +10322,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_load( ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10349,6 +10499,9 @@ define amdgpu_kernel void @global_singlethread_one_as_unordered_store( ; ; GFX7-LABEL: global_singlethread_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10502,6 +10655,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_store( ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10655,6 +10811,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_store( ; ; GFX7-LABEL: global_singlethread_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10808,6 +10967,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_store( ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10960,6 +11122,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11111,6 +11276,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11262,6 +11430,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11413,6 +11584,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11564,6 +11738,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11717,6 +11894,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11897,6 +12077,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12077,6 +12260,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12260,6 +12446,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_monotonic_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12476,6 +12665,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12692,6 +12884,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12908,6 +13103,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13124,6 +13322,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13340,6 +13541,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13556,6 +13760,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13772,6 +13979,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13988,6 +14198,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14204,6 +14417,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14420,6 +14636,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14636,6 +14855,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14852,6 +15074,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15068,6 +15293,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15284,6 +15512,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15503,6 +15734,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_monotonic_ret_cm ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15751,6 +15985,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_monotonic_ret_cmpx ; ; GFX7-LABEL: global_singlethread_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15999,6 +16236,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_monotonic_ret_cmpx ; ; GFX7-LABEL: global_singlethread_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16247,6 +16487,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_monotonic_ret_cmpx ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16495,6 +16738,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_monotonic_ret_cmpx ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16743,6 +16989,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_acquire_ret_cmpx ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16991,6 +17240,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_acquire_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17239,6 +17491,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_acquire_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17487,6 +17742,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_acquire_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17735,6 +17993,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_acquire_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17983,6 +18244,9 @@ define amdgpu_kernel void @global_singlethread_one_as_monotonic_seq_cst_ret_cmpx ; ; GFX7-LABEL: global_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18231,6 +18495,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acquire_seq_cst_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18479,6 +18746,9 @@ define amdgpu_kernel void @global_singlethread_one_as_release_seq_cst_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18727,6 +18997,9 @@ define amdgpu_kernel void @global_singlethread_one_as_acq_rel_seq_cst_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18975,6 +19248,9 @@ define amdgpu_kernel void @global_singlethread_one_as_seq_cst_seq_cst_ret_cmpxch ; ; GFX7-LABEL: global_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-system.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-system.ll index 69404247ccd6e..9c11781da56f2 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-system.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-system.ll @@ -41,6 +41,9 @@ define amdgpu_kernel void @global_system_unordered_load( ; ; GFX7-LABEL: global_system_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -222,6 +225,9 @@ define amdgpu_kernel void @global_system_monotonic_load( ; ; GFX7-LABEL: global_system_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -404,6 +410,9 @@ define amdgpu_kernel void @global_system_acquire_load( ; ; GFX7-LABEL: global_system_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -604,6 +613,9 @@ define amdgpu_kernel void @global_system_seq_cst_load( ; ; GFX7-LABEL: global_system_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -813,6 +825,9 @@ define amdgpu_kernel void @global_system_unordered_store( ; ; GFX7-LABEL: global_system_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -966,6 +981,9 @@ define amdgpu_kernel void @global_system_monotonic_store( ; ; GFX7-LABEL: global_system_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1120,6 +1138,9 @@ define amdgpu_kernel void @global_system_release_store( ; ; GFX7-LABEL: global_system_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1302,6 +1323,9 @@ define amdgpu_kernel void @global_system_seq_cst_store( ; ; GFX7-LABEL: global_system_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1482,6 +1506,9 @@ define amdgpu_kernel void @global_system_monotonic_atomicrmw( ; ; GFX7-LABEL: global_system_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1635,6 +1662,9 @@ define amdgpu_kernel void @global_system_acquire_atomicrmw( ; ; GFX7-LABEL: global_system_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1816,6 +1846,9 @@ define amdgpu_kernel void @global_system_release_atomicrmw( ; ; GFX7-LABEL: global_system_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1998,6 +2031,9 @@ define amdgpu_kernel void @global_system_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_system_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2209,6 +2245,9 @@ define amdgpu_kernel void @global_system_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_system_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2420,6 +2459,9 @@ define amdgpu_kernel void @global_system_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_system_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2619,6 +2661,9 @@ define amdgpu_kernel void @global_system_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_system_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2850,6 +2895,9 @@ define amdgpu_kernel void @global_system_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_system_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -3082,6 +3130,9 @@ define amdgpu_kernel void @global_system_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3300,6 +3351,9 @@ define amdgpu_kernel void @global_system_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3546,6 +3600,9 @@ define amdgpu_kernel void @global_system_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3793,6 +3850,9 @@ define amdgpu_kernel void @global_system_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4069,6 +4129,9 @@ define amdgpu_kernel void @global_system_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4344,6 +4407,9 @@ define amdgpu_kernel void @global_system_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4591,6 +4657,9 @@ define amdgpu_kernel void @global_system_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4839,6 +4908,9 @@ define amdgpu_kernel void @global_system_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5115,6 +5187,9 @@ define amdgpu_kernel void @global_system_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5391,6 +5466,9 @@ define amdgpu_kernel void @global_system_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5667,6 +5745,9 @@ define amdgpu_kernel void @global_system_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_system_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5943,6 +6024,9 @@ define amdgpu_kernel void @global_system_monotonic_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6192,6 +6276,9 @@ define amdgpu_kernel void @global_system_acquire_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6460,6 +6547,9 @@ define amdgpu_kernel void @global_system_acq_rel_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6760,6 +6850,9 @@ define amdgpu_kernel void @global_system_seq_cst_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7059,6 +7152,9 @@ define amdgpu_kernel void @global_system_monotonic_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7330,6 +7426,9 @@ define amdgpu_kernel void @global_system_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7598,6 +7697,9 @@ define amdgpu_kernel void @global_system_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7898,6 +8000,9 @@ define amdgpu_kernel void @global_system_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8198,6 +8303,9 @@ define amdgpu_kernel void @global_system_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8498,6 +8606,9 @@ define amdgpu_kernel void @global_system_monotonic_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8798,6 +8909,9 @@ define amdgpu_kernel void @global_system_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9094,6 +9208,9 @@ define amdgpu_kernel void @global_system_relese_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_relese_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9394,6 +9511,9 @@ define amdgpu_kernel void @global_system_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9694,6 +9814,9 @@ define amdgpu_kernel void @global_system_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9992,6 +10115,9 @@ define amdgpu_kernel void @global_system_one_as_unordered_load( ; ; GFX7-LABEL: global_system_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10173,6 +10299,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_load( ; ; GFX7-LABEL: global_system_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10355,6 +10484,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_load( ; ; GFX7-LABEL: global_system_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10555,6 +10687,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_load( ; ; GFX7-LABEL: global_system_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10764,6 +10899,9 @@ define amdgpu_kernel void @global_system_one_as_unordered_store( ; ; GFX7-LABEL: global_system_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10917,6 +11055,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_store( ; ; GFX7-LABEL: global_system_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11071,6 +11212,9 @@ define amdgpu_kernel void @global_system_one_as_release_store( ; ; GFX7-LABEL: global_system_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11253,6 +11397,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_store( ; ; GFX7-LABEL: global_system_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11433,6 +11580,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11586,6 +11736,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11767,6 +11920,9 @@ define amdgpu_kernel void @global_system_one_as_release_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11949,6 +12105,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12160,6 +12319,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12371,6 +12533,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12570,6 +12735,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12801,6 +12969,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_system_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13033,6 +13204,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13251,6 +13425,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13497,6 +13674,9 @@ define amdgpu_kernel void @global_system_one_as_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13744,6 +13924,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14020,6 +14203,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14295,6 +14481,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14542,6 +14731,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14790,6 +14982,9 @@ define amdgpu_kernel void @global_system_one_as_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15066,6 +15261,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15342,6 +15540,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15618,6 +15819,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15894,6 +16098,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16170,6 +16377,9 @@ define amdgpu_kernel void @global_system_one_as_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16446,6 +16656,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16722,6 +16935,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16998,6 +17214,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17247,6 +17466,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17514,6 +17736,9 @@ define amdgpu_kernel void @global_system_one_as_release_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17792,6 +18017,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18092,6 +18320,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18391,6 +18622,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18662,6 +18896,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18930,6 +19167,9 @@ define amdgpu_kernel void @global_system_one_as_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19230,6 +19470,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19530,6 +19773,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19830,6 +20076,9 @@ define amdgpu_kernel void @global_system_one_as_monotonic_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20130,6 +20379,9 @@ define amdgpu_kernel void @global_system_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20426,6 +20678,9 @@ define amdgpu_kernel void @global_system_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20726,6 +20981,9 @@ define amdgpu_kernel void @global_system_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -21026,6 +21284,9 @@ define amdgpu_kernel void @global_system_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_system_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-volatile.ll index 7dfd5e60c24f8..8a5c5dda9f79c 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-volatile.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-volatile.ll @@ -37,6 +37,9 @@ define amdgpu_kernel void @global_volatile_load_0( ; ; GFX7-LABEL: global_volatile_load_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -184,6 +187,9 @@ define amdgpu_kernel void @global_volatile_load_1( ; ; GFX7-LABEL: global_volatile_load_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[4:5], 0x2 @@ -372,6 +378,9 @@ define amdgpu_kernel void @global_volatile_store_0( ; ; GFX7-LABEL: global_volatile_store_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -527,6 +536,9 @@ define amdgpu_kernel void @global_volatile_store_1( ; ; GFX7-LABEL: global_volatile_store_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -718,6 +730,9 @@ define amdgpu_kernel void @global_volatile_workgroup_acquire_load( ; ; GFX7-LABEL: global_volatile_workgroup_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -852,6 +867,9 @@ define amdgpu_kernel void @global_volatile_workgroup_release_store( ; ; GFX7-LABEL: global_volatile_workgroup_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-wavefront.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-wavefront.ll index 4b6c99282dc13..151ba07a0b531 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-wavefront.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-wavefront.ll @@ -41,6 +41,9 @@ define amdgpu_kernel void @global_wavefront_unordered_load( ; ; GFX7-LABEL: global_wavefront_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -222,6 +225,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_load( ; ; GFX7-LABEL: global_wavefront_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -403,6 +409,9 @@ define amdgpu_kernel void @global_wavefront_acquire_load( ; ; GFX7-LABEL: global_wavefront_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -584,6 +593,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_load( ; ; GFX7-LABEL: global_wavefront_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -758,6 +770,9 @@ define amdgpu_kernel void @global_wavefront_unordered_store( ; ; GFX7-LABEL: global_wavefront_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -911,6 +926,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_store( ; ; GFX7-LABEL: global_wavefront_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1064,6 +1082,9 @@ define amdgpu_kernel void @global_wavefront_release_store( ; ; GFX7-LABEL: global_wavefront_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1217,6 +1238,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_store( ; ; GFX7-LABEL: global_wavefront_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1369,6 +1393,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_atomicrmw( ; ; GFX7-LABEL: global_wavefront_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1520,6 +1547,9 @@ define amdgpu_kernel void @global_wavefront_acquire_atomicrmw( ; ; GFX7-LABEL: global_wavefront_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1671,6 +1701,9 @@ define amdgpu_kernel void @global_wavefront_release_atomicrmw( ; ; GFX7-LABEL: global_wavefront_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1822,6 +1855,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_wavefront_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1973,6 +2009,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_wavefront_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2126,6 +2165,9 @@ define amdgpu_kernel void @global_wavefront_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_wavefront_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2306,6 +2348,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_wavefront_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2486,6 +2531,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_wavefront_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2669,6 +2717,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -2885,6 +2936,9 @@ define amdgpu_kernel void @global_wavefront_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3101,6 +3155,9 @@ define amdgpu_kernel void @global_wavefront_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3317,6 +3374,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3533,6 +3593,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3749,6 +3812,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3965,6 +4031,9 @@ define amdgpu_kernel void @global_wavefront_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4181,6 +4250,9 @@ define amdgpu_kernel void @global_wavefront_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4397,6 +4469,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4613,6 +4688,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4829,6 +4907,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5045,6 +5126,9 @@ define amdgpu_kernel void @global_wavefront_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5261,6 +5345,9 @@ define amdgpu_kernel void @global_wavefront_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5477,6 +5564,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5693,6 +5783,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5912,6 +6005,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6160,6 +6256,9 @@ define amdgpu_kernel void @global_wavefront_acquire_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6408,6 +6507,9 @@ define amdgpu_kernel void @global_wavefront_release_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6656,6 +6758,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6904,6 +7009,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7152,6 +7260,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7400,6 +7511,9 @@ define amdgpu_kernel void @global_wavefront_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7648,6 +7762,9 @@ define amdgpu_kernel void @global_wavefront_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7896,6 +8013,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8144,6 +8264,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8392,6 +8515,9 @@ define amdgpu_kernel void @global_wavefront_monotonic_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8640,6 +8766,9 @@ define amdgpu_kernel void @global_wavefront_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8888,6 +9017,9 @@ define amdgpu_kernel void @global_wavefront_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9136,6 +9268,9 @@ define amdgpu_kernel void @global_wavefront_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9384,6 +9519,9 @@ define amdgpu_kernel void @global_wavefront_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9632,6 +9770,9 @@ define amdgpu_kernel void @global_wavefront_one_as_unordered_load( ; ; GFX7-LABEL: global_wavefront_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9813,6 +9954,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_load( ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -9994,6 +10138,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_load( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10175,6 +10322,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_load( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10349,6 +10499,9 @@ define amdgpu_kernel void @global_wavefront_one_as_unordered_store( ; ; GFX7-LABEL: global_wavefront_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10502,6 +10655,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_store( ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10655,6 +10811,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_store( ; ; GFX7-LABEL: global_wavefront_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10808,6 +10967,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_store( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10960,6 +11122,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11111,6 +11276,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11262,6 +11430,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11413,6 +11584,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11564,6 +11738,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11717,6 +11894,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11897,6 +12077,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12077,6 +12260,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12260,6 +12446,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12476,6 +12665,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12692,6 +12884,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -12908,6 +13103,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13124,6 +13322,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13340,6 +13541,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13556,6 +13760,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13772,6 +13979,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13988,6 +14198,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14204,6 +14417,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14420,6 +14636,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14636,6 +14855,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14852,6 +15074,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15068,6 +15293,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15284,6 +15512,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15503,6 +15734,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_monotonic_ret_cmpxc ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15751,6 +15985,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_wavefront_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -15999,6 +16236,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_wavefront_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16247,6 +16487,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16495,6 +16738,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16743,6 +16989,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_acquire_ret_cmpxchg ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16991,6 +17240,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17239,6 +17491,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17487,6 +17742,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17735,6 +17993,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17983,6 +18244,9 @@ define amdgpu_kernel void @global_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg ; ; GFX7-LABEL: global_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18231,6 +18495,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18479,6 +18746,9 @@ define amdgpu_kernel void @global_wavefront_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18727,6 +18997,9 @@ define amdgpu_kernel void @global_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18975,6 +19248,9 @@ define amdgpu_kernel void @global_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-workgroup.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-workgroup.ll index 46d65187cb1b2..69b0c7f93ab0e 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-workgroup.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-global-workgroup.ll @@ -41,6 +41,9 @@ define amdgpu_kernel void @global_workgroup_unordered_load( ; ; GFX7-LABEL: global_workgroup_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -222,6 +225,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_load( ; ; GFX7-LABEL: global_workgroup_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -403,6 +409,9 @@ define amdgpu_kernel void @global_workgroup_acquire_load( ; ; GFX7-LABEL: global_workgroup_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -590,6 +599,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_load( ; ; GFX7-LABEL: global_workgroup_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -780,6 +792,9 @@ define amdgpu_kernel void @global_workgroup_unordered_store( ; ; GFX7-LABEL: global_workgroup_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -933,6 +948,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_store( ; ; GFX7-LABEL: global_workgroup_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1087,6 +1105,9 @@ define amdgpu_kernel void @global_workgroup_release_store( ; ; GFX7-LABEL: global_workgroup_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1258,6 +1279,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_store( ; ; GFX7-LABEL: global_workgroup_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1427,6 +1451,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_atomicrmw( ; ; GFX7-LABEL: global_workgroup_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1578,6 +1605,9 @@ define amdgpu_kernel void @global_workgroup_acquire_atomicrmw( ; ; GFX7-LABEL: global_workgroup_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1740,6 +1770,9 @@ define amdgpu_kernel void @global_workgroup_release_atomicrmw( ; ; GFX7-LABEL: global_workgroup_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -1909,6 +1942,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_workgroup_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2088,6 +2124,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_workgroup_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2268,6 +2307,9 @@ define amdgpu_kernel void @global_workgroup_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_workgroup_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2454,6 +2496,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_workgroup_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2659,6 +2704,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_workgroup_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -2866,6 +2914,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3082,6 +3133,9 @@ define amdgpu_kernel void @global_workgroup_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3309,6 +3363,9 @@ define amdgpu_kernel void @global_workgroup_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3543,6 +3600,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -3787,6 +3847,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4030,6 +4093,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4256,6 +4322,9 @@ define amdgpu_kernel void @global_workgroup_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4483,6 +4552,9 @@ define amdgpu_kernel void @global_workgroup_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4727,6 +4799,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -4971,6 +5046,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5215,6 +5293,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5459,6 +5540,9 @@ define amdgpu_kernel void @global_workgroup_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5703,6 +5787,9 @@ define amdgpu_kernel void @global_workgroup_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -5947,6 +6034,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6191,6 +6281,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -6437,6 +6530,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6685,6 +6781,9 @@ define amdgpu_kernel void @global_workgroup_acquire_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -6939,6 +7038,9 @@ define amdgpu_kernel void @global_workgroup_release_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7205,6 +7307,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7478,6 +7583,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_monotonic_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -7750,6 +7858,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8005,6 +8116,9 @@ define amdgpu_kernel void @global_workgroup_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8259,6 +8373,9 @@ define amdgpu_kernel void @global_workgroup_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8532,6 +8649,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -8805,6 +8925,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9078,6 +9201,9 @@ define amdgpu_kernel void @global_workgroup_monotonic_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9351,6 +9477,9 @@ define amdgpu_kernel void @global_workgroup_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9622,6 +9751,9 @@ define amdgpu_kernel void @global_workgroup_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -9895,6 +10027,9 @@ define amdgpu_kernel void @global_workgroup_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10168,6 +10303,9 @@ define amdgpu_kernel void @global_workgroup_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -10440,6 +10578,9 @@ define amdgpu_kernel void @global_workgroup_one_as_unordered_load( ; ; GFX7-LABEL: global_workgroup_one_as_unordered_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10621,6 +10762,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_load( ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10802,6 +10946,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_load( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -10988,6 +11135,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_load( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11175,6 +11325,9 @@ define amdgpu_kernel void @global_workgroup_one_as_unordered_store( ; ; GFX7-LABEL: global_workgroup_one_as_unordered_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11328,6 +11481,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_store( ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11481,6 +11637,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_store( ; ; GFX7-LABEL: global_workgroup_one_as_release_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11644,6 +11803,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_store( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_store: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11806,6 +11968,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -11957,6 +12122,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12118,6 +12286,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_release_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12279,6 +12450,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12450,6 +12624,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12623,6 +12800,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_ret_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -12808,6 +12988,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_ret_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13005,6 +13188,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_ret_atomicrmw( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_ret_atomicrmw: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x2 ; GFX7-NEXT: s_waitcnt lgkmcnt(0) @@ -13205,6 +13391,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13421,6 +13610,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13647,6 +13839,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_release_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -13873,6 +14068,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14109,6 +14307,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_monotonic_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_monotonic_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14345,6 +14546,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14571,6 +14775,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -14797,6 +15004,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_release_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15033,6 +15243,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15269,6 +15482,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_acquire_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_acquire_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15505,6 +15721,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15741,6 +15960,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -15977,6 +16199,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_release_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16213,6 +16438,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16449,6 +16677,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_seq_cst_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_seq_cst_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[4:5], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX7-NEXT: s_load_dword s7, s[4:5], 0x2 @@ -16688,6 +16919,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_monotonic_ret_cmpxc ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -16936,6 +17170,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_workgroup_one_as_acquire_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17189,6 +17426,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_workgroup_one_as_release_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17447,6 +17687,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17712,6 +17955,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -17977,6 +18223,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_acquire_ret_cmpxchg ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18232,6 +18481,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18485,6 +18737,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_release_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -18750,6 +19005,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19015,6 +19273,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_acquire_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_acquire_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19280,6 +19541,9 @@ define amdgpu_kernel void @global_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg ; ; GFX7-LABEL: global_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19545,6 +19809,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acquire_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acquire_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -19808,6 +20075,9 @@ define amdgpu_kernel void @global_workgroup_one_as_release_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_release_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20073,6 +20343,9 @@ define amdgpu_kernel void @global_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 @@ -20338,6 +20611,9 @@ define amdgpu_kernel void @global_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg( ; ; GFX7-LABEL: global_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_mov_b64 s[6:7], s[8:9] ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[6:7], 0x0 ; GFX7-NEXT: s_load_dword s9, s[6:7], 0x2 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-nontemporal.ll index 04b0f00fe77b5..78209ee34cad4 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-nontemporal.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-nontemporal.ll @@ -38,6 +38,9 @@ define amdgpu_kernel void @local_nontemporal_load_0( ; ; GFX7-LABEL: local_nontemporal_load_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_mov_b32 m0, -1 @@ -224,6 +227,9 @@ define amdgpu_kernel void @local_nontemporal_load_1( ; ; GFX7-LABEL: local_nontemporal_load_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_mov_b32 s7, 2 @@ -830,6 +836,9 @@ define amdgpu_kernel void @local_nontemporal_volatile_load( ; ; GFX7-LABEL: local_nontemporal_volatile_load: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_mov_b32 m0, -1 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-volatile.ll index 9e5f5fcffca9f..bc2508411ed6b 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-volatile.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-local-volatile.ll @@ -34,6 +34,9 @@ define amdgpu_kernel void @local_volatile_load_0( ; ; GFX7-LABEL: local_volatile_load_0: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_mov_b32 m0, -1 @@ -172,6 +175,9 @@ define amdgpu_kernel void @local_volatile_load_1( ; ; GFX7-LABEL: local_volatile_load_1: ; GFX7: ; %bb.0: ; %entry +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 ; GFX7-NEXT: s_mov_b32 s7, 2 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll index fceee413f3f97..2aa4f021c259c 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll @@ -38,7 +38,10 @@ define amdgpu_kernel void @private_nontemporal_load_0( ; ; GFX7-LABEL: private_nontemporal_load_0: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 @@ -53,7 +56,7 @@ define amdgpu_kernel void @private_nontemporal_load_0( ; ; GFX10-WGP-LABEL: private_nontemporal_load_0: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -67,7 +70,7 @@ define amdgpu_kernel void @private_nontemporal_load_0( ; ; GFX10-CU-LABEL: private_nontemporal_load_0: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -107,7 +110,7 @@ define amdgpu_kernel void @private_nontemporal_load_0( ; ; GFX90A-NOTTGSPLIT-LABEL: private_nontemporal_load_0: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry -; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -121,7 +124,7 @@ define amdgpu_kernel void @private_nontemporal_load_0( ; ; GFX90A-TGSPLIT-LABEL: private_nontemporal_load_0: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry -; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-TGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -232,7 +235,10 @@ define amdgpu_kernel void @private_nontemporal_load_1( ; ; GFX7-LABEL: private_nontemporal_load_1: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 @@ -249,7 +255,7 @@ define amdgpu_kernel void @private_nontemporal_load_1( ; ; GFX10-WGP-LABEL: private_nontemporal_load_1: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: v_mov_b32_e32 v1, v0 ; GFX10-WGP-NEXT: s_load_dword s7, s[8:9], 0x0 @@ -265,7 +271,7 @@ define amdgpu_kernel void @private_nontemporal_load_1( ; ; GFX10-CU-LABEL: private_nontemporal_load_1: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: v_mov_b32_e32 v1, v0 ; GFX10-CU-NEXT: s_load_dword s7, s[8:9], 0x0 @@ -309,7 +315,7 @@ define amdgpu_kernel void @private_nontemporal_load_1( ; ; GFX90A-NOTTGSPLIT-LABEL: private_nontemporal_load_1: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry -; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-NOTTGSPLIT-NEXT: v_mov_b32_e32 v1, v0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x0 @@ -328,7 +334,7 @@ define amdgpu_kernel void @private_nontemporal_load_1( ; ; GFX90A-TGSPLIT-LABEL: private_nontemporal_load_1: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry -; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-TGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-TGSPLIT-NEXT: v_mov_b32_e32 v1, v0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s7, s[8:9], 0x0 @@ -470,7 +476,7 @@ define amdgpu_kernel void @private_nontemporal_store_0( ; ; GFX7-LABEL: private_nontemporal_store_0: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 @@ -484,7 +490,7 @@ define amdgpu_kernel void @private_nontemporal_store_0( ; ; GFX10-WGP-LABEL: private_nontemporal_store_0: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 @@ -498,7 +504,7 @@ define amdgpu_kernel void @private_nontemporal_store_0( ; ; GFX10-CU-LABEL: private_nontemporal_store_0: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 @@ -530,7 +536,7 @@ define amdgpu_kernel void @private_nontemporal_store_0( ; ; GFX90A-NOTTGSPLIT-LABEL: private_nontemporal_store_0: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry -; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 @@ -544,7 +550,7 @@ define amdgpu_kernel void @private_nontemporal_store_0( ; ; GFX90A-TGSPLIT-LABEL: private_nontemporal_store_0: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry -; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-TGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s4, s[8:9], 0x8 @@ -647,7 +653,7 @@ define amdgpu_kernel void @private_nontemporal_store_1( ; ; GFX7-LABEL: private_nontemporal_store_1: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s5, s[8:9], 0x2 @@ -663,7 +669,7 @@ define amdgpu_kernel void @private_nontemporal_store_1( ; ; GFX10-WGP-LABEL: private_nontemporal_store_1: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 @@ -678,7 +684,7 @@ define amdgpu_kernel void @private_nontemporal_store_1( ; ; GFX10-CU-LABEL: private_nontemporal_store_1: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 @@ -713,7 +719,7 @@ define amdgpu_kernel void @private_nontemporal_store_1( ; ; GFX90A-NOTTGSPLIT-LABEL: private_nontemporal_store_1: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry -; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 @@ -731,7 +737,7 @@ define amdgpu_kernel void @private_nontemporal_store_1( ; ; GFX90A-TGSPLIT-LABEL: private_nontemporal_store_1: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry -; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-TGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x8 @@ -874,7 +880,10 @@ define amdgpu_kernel void @private_nontemporal_volatile_load( ; ; GFX7-LABEL: private_nontemporal_volatile_load: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 @@ -889,7 +898,7 @@ define amdgpu_kernel void @private_nontemporal_volatile_load( ; ; GFX10-WGP-LABEL: private_nontemporal_volatile_load: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -903,7 +912,7 @@ define amdgpu_kernel void @private_nontemporal_volatile_load( ; ; GFX10-CU-LABEL: private_nontemporal_volatile_load: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -943,7 +952,7 @@ define amdgpu_kernel void @private_nontemporal_volatile_load( ; ; GFX90A-NOTTGSPLIT-LABEL: private_nontemporal_volatile_load: ; GFX90A-NOTTGSPLIT: ; %bb.0: ; %entry -; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-NOTTGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-NOTTGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX90A-NOTTGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -957,7 +966,7 @@ define amdgpu_kernel void @private_nontemporal_volatile_load( ; ; GFX90A-TGSPLIT-LABEL: private_nontemporal_volatile_load: ; GFX90A-TGSPLIT: ; %bb.0: ; %entry -; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s15 +; GFX90A-TGSPLIT-NEXT: s_add_u32 s0, s0, s17 ; GFX90A-TGSPLIT-NEXT: s_addc_u32 s1, s1, 0 ; GFX90A-TGSPLIT-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX90A-TGSPLIT-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll index f8fb7986938f2..df4193969f8a0 100644 --- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll +++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll @@ -38,7 +38,10 @@ define amdgpu_kernel void @private_volatile_load_0( ; ; GFX7-LABEL: private_volatile_load_0: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 @@ -53,7 +56,7 @@ define amdgpu_kernel void @private_volatile_load_0( ; ; GFX10-WGP-LABEL: private_volatile_load_0: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -67,7 +70,7 @@ define amdgpu_kernel void @private_volatile_load_0( ; ; GFX10-CU-LABEL: private_volatile_load_0: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x8 @@ -190,7 +193,10 @@ define amdgpu_kernel void @private_volatile_load_1( ; ; GFX7-LABEL: private_volatile_load_1: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX7-NEXT: s_add_i32 s12, s12, s17 +; GFX7-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dword s6, s[8:9], 0x0 ; GFX7-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x2 @@ -207,7 +213,7 @@ define amdgpu_kernel void @private_volatile_load_1( ; ; GFX10-WGP-LABEL: private_volatile_load_1: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: v_mov_b32_e32 v1, v0 ; GFX10-WGP-NEXT: s_load_dword s7, s[8:9], 0x0 @@ -223,7 +229,7 @@ define amdgpu_kernel void @private_volatile_load_1( ; ; GFX10-CU-LABEL: private_volatile_load_1: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: v_mov_b32_e32 v1, v0 ; GFX10-CU-NEXT: s_load_dword s7, s[8:9], 0x0 @@ -365,7 +371,7 @@ define amdgpu_kernel void @private_volatile_store_0( ; ; GFX7-LABEL: private_volatile_store_0: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s4, s[8:9], 0x2 @@ -380,7 +386,7 @@ define amdgpu_kernel void @private_volatile_store_0( ; ; GFX10-WGP-LABEL: private_volatile_store_0: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s4, s[8:9], 0x8 @@ -395,7 +401,7 @@ define amdgpu_kernel void @private_volatile_store_0( ; ; GFX10-CU-LABEL: private_volatile_store_0: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s4, s[8:9], 0x8 @@ -515,7 +521,7 @@ define amdgpu_kernel void @private_volatile_store_1( ; ; GFX7-LABEL: private_volatile_store_1: ; GFX7: ; %bb.0: ; %entry -; GFX7-NEXT: s_add_u32 s0, s0, s15 +; GFX7-NEXT: s_add_u32 s0, s0, s17 ; GFX7-NEXT: s_addc_u32 s1, s1, 0 ; GFX7-NEXT: s_load_dwordx2 s[6:7], s[8:9], 0x0 ; GFX7-NEXT: s_load_dword s5, s[8:9], 0x2 @@ -532,7 +538,7 @@ define amdgpu_kernel void @private_volatile_store_1( ; ; GFX10-WGP-LABEL: private_volatile_store_1: ; GFX10-WGP: ; %bb.0: ; %entry -; GFX10-WGP-NEXT: s_add_u32 s0, s0, s15 +; GFX10-WGP-NEXT: s_add_u32 s0, s0, s17 ; GFX10-WGP-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-WGP-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-WGP-NEXT: s_load_dword s6, s[8:9], 0x8 @@ -548,7 +554,7 @@ define amdgpu_kernel void @private_volatile_store_1( ; ; GFX10-CU-LABEL: private_volatile_store_1: ; GFX10-CU: ; %bb.0: ; %entry -; GFX10-CU-NEXT: s_add_u32 s0, s0, s15 +; GFX10-CU-NEXT: s_add_u32 s0, s0, s17 ; GFX10-CU-NEXT: s_addc_u32 s1, s1, 0 ; GFX10-CU-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 ; GFX10-CU-NEXT: s_load_dword s6, s[8:9], 0x8 diff --git a/llvm/test/CodeGen/AMDGPU/min.ll b/llvm/test/CodeGen/AMDGPU/min.ll index aaf81e2fa4000..07072f6a36296 100644 --- a/llvm/test/CodeGen/AMDGPU/min.ll +++ b/llvm/test/CodeGen/AMDGPU/min.ll @@ -34,10 +34,13 @@ define amdgpu_kernel void @v_test_imin_sle_i32(ptr addrspace(1) %out, ptr addrsp ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: v_add_i32_e32 v2, vcc, s4, v4 ; CI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -56,10 +59,13 @@ define amdgpu_kernel void @v_test_imin_sle_i32(ptr addrspace(1) %out, ptr addrsp ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -144,6 +150,9 @@ define amdgpu_kernel void @s_test_imin_sle_i32(ptr addrspace(1) %out, i32 %a, i3 ; CI-LABEL: s_test_imin_sle_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_i32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -155,6 +164,9 @@ define amdgpu_kernel void @s_test_imin_sle_i32(ptr addrspace(1) %out, i32 %a, i3 ; VI-LABEL: s_test_imin_sle_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_i32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -214,6 +226,9 @@ define amdgpu_kernel void @s_test_imin_sle_v1i32(ptr addrspace(1) %out, <1 x i32 ; CI-LABEL: s_test_imin_sle_v1i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_i32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -225,6 +240,9 @@ define amdgpu_kernel void @s_test_imin_sle_v1i32(ptr addrspace(1) %out, <1 x i32 ; VI-LABEL: s_test_imin_sle_v1i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_i32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -288,6 +306,9 @@ define amdgpu_kernel void @s_test_imin_sle_v4i32(ptr addrspace(1) %out, <4 x i32 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x4 ; CI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_i32 s3, s3, s7 ; CI-NEXT: s_min_i32 s2, s2, s6 @@ -306,6 +327,9 @@ define amdgpu_kernel void @s_test_imin_sle_v4i32(ptr addrspace(1) %out, <4 x i32 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x10 ; VI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_i32 s3, s3, s7 ; VI-NEXT: s_min_i32 s2, s2, s6 @@ -414,11 +438,14 @@ define amdgpu_kernel void @s_test_imin_sle_i8(ptr addrspace(1) %out, [8 x i32], ; CI-NEXT: s_load_dword s2, s[8:9], 0xa ; CI-NEXT: s_load_dword s3, s[8:9], 0x13 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_sext_i32_i8 s2, s2 ; CI-NEXT: s_sext_i32_i8 s3, s3 ; CI-NEXT: s_min_i32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: flat_store_byte v[0:1], v2 @@ -429,11 +456,14 @@ define amdgpu_kernel void @s_test_imin_sle_i8(ptr addrspace(1) %out, [8 x i32], ; VI-NEXT: s_load_dword s2, s[8:9], 0x28 ; VI-NEXT: s_load_dword s3, s[8:9], 0x4c ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_sext_i32_i8 s2, s2 ; VI-NEXT: s_sext_i32_i8 s3, s3 ; VI-NEXT: s_min_i32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: flat_store_byte v[0:1], v2 @@ -549,6 +579,8 @@ define amdgpu_kernel void @s_test_imin_sle_v4i8(ptr addrspace(1) %out, [8 x i32] ; CI-NEXT: s_load_dword s2, s[8:9], 0xa ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: s_load_dword s3, s[8:9], 0x13 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_ashr_i32 s4, s2, 24 ; CI-NEXT: s_sext_i32_i8 s5, s2 @@ -572,6 +604,7 @@ define amdgpu_kernel void @s_test_imin_sle_v4i8(ptr addrspace(1) %out, [8 x i32] ; CI-NEXT: s_and_b32 s3, s3, 0xffff ; CI-NEXT: s_or_b32 s2, s3, s2 ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: flat_store_dword v[0:1], v2 @@ -582,6 +615,8 @@ define amdgpu_kernel void @s_test_imin_sle_v4i8(ptr addrspace(1) %out, [8 x i32] ; VI-NEXT: s_load_dword s2, s[8:9], 0x28 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: s_load_dword s3, s[8:9], 0x4c +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_ashr_i32 s4, s2, 24 ; VI-NEXT: s_bfe_i32 s5, s2, 0x80010 @@ -605,6 +640,7 @@ define amdgpu_kernel void @s_test_imin_sle_v4i8(ptr addrspace(1) %out, [8 x i32] ; VI-NEXT: s_and_b32 s2, s2, 0xffff ; VI-NEXT: s_or_b32 s2, s2, s4 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: flat_store_dword v[0:1], v2 @@ -757,6 +793,9 @@ define amdgpu_kernel void @s_test_imin_sle_v2i16(ptr addrspace(1) %out, <2 x i16 ; CI-LABEL: s_test_imin_sle_v2i16: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_ashr_i32 s4, s2, 16 ; CI-NEXT: s_sext_i32_i16 s2, s2 @@ -776,6 +815,9 @@ define amdgpu_kernel void @s_test_imin_sle_v2i16(ptr addrspace(1) %out, <2 x i16 ; VI-LABEL: s_test_imin_sle_v2i16: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_ashr_i32 s4, s2, 16 ; VI-NEXT: s_sext_i32_i16 s2, s2 @@ -857,6 +899,9 @@ define amdgpu_kernel void @s_test_imin_sle_v4i16(ptr addrspace(1) %out, <4 x i16 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_ashr_i32 s6, s0, 16 ; CI-NEXT: s_ashr_i32 s7, s1, 16 @@ -887,6 +932,9 @@ define amdgpu_kernel void @s_test_imin_sle_v4i16(ptr addrspace(1) %out, <4 x i16 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_ashr_i32 s6, s1, 16 ; VI-NEXT: s_sext_i32_i16 s1, s1 @@ -983,10 +1031,13 @@ define amdgpu_kernel void @v_test_imin_slt_i32(ptr addrspace(1) %out, ptr addrsp ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: v_add_i32_e32 v2, vcc, s4, v4 ; CI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -1005,10 +1056,13 @@ define amdgpu_kernel void @v_test_imin_slt_i32(ptr addrspace(1) %out, ptr addrsp ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -1122,10 +1176,13 @@ define amdgpu_kernel void @v_test_imin_slt_i16(ptr addrspace(1) %out, ptr addrsp ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 1, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: v_add_i32_e32 v2, vcc, s4, v4 ; CI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -1144,10 +1201,13 @@ define amdgpu_kernel void @v_test_imin_slt_i16(ptr addrspace(1) %out, ptr addrsp ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 1, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -1233,6 +1293,9 @@ define amdgpu_kernel void @s_test_imin_slt_i32(ptr addrspace(1) %out, i32 %a, i3 ; CI-LABEL: s_test_imin_slt_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_i32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -1244,6 +1307,9 @@ define amdgpu_kernel void @s_test_imin_slt_i32(ptr addrspace(1) %out, i32 %a, i3 ; VI-LABEL: s_test_imin_slt_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_i32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -1305,6 +1371,9 @@ define amdgpu_kernel void @s_test_imin_slt_v2i32(ptr addrspace(1) %out, <2 x i32 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_i32 s1, s1, s3 ; CI-NEXT: s_min_i32 s0, s0, s2 @@ -1319,6 +1388,9 @@ define amdgpu_kernel void @s_test_imin_slt_v2i32(ptr addrspace(1) %out, <2 x i32 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_i32 s1, s1, s3 ; VI-NEXT: s_min_i32 s0, s0, s2 @@ -1391,6 +1463,9 @@ define amdgpu_kernel void @s_test_imin_slt_imm_i32(ptr addrspace(1) %out, i32 %a ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_i32 s2, s2, 8 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -1403,6 +1478,9 @@ define amdgpu_kernel void @s_test_imin_slt_imm_i32(ptr addrspace(1) %out, i32 %a ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_i32 s2, s2, 8 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -1468,6 +1546,9 @@ define amdgpu_kernel void @s_test_imin_sle_imm_i32(ptr addrspace(1) %out, i32 %a ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_i32 s2, s2, 8 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -1480,6 +1561,9 @@ define amdgpu_kernel void @s_test_imin_sle_imm_i32(ptr addrspace(1) %out, i32 %a ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_i32 s2, s2, 8 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -1557,10 +1641,13 @@ define amdgpu_kernel void @v_test_umin_ule_i32(ptr addrspace(1) %out, ptr addrsp ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: v_add_i32_e32 v2, vcc, s4, v4 ; CI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -1579,10 +1666,13 @@ define amdgpu_kernel void @v_test_umin_ule_i32(ptr addrspace(1) %out, ptr addrsp ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -1686,12 +1776,15 @@ define amdgpu_kernel void @v_test_umin_ule_v3i32(ptr addrspace(1) %out, ptr addr ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v6, 4, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v6 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_mov_b32_e32 v2, s5 ; CI-NEXT: v_add_i32_e32 v3, vcc, s4, v6 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v4, vcc, 0, v2, vcc ; CI-NEXT: flat_load_dwordx3 v[0:2], v[0:1] ; CI-NEXT: flat_load_dwordx3 v[3:5], v[3:4] @@ -1710,12 +1803,15 @@ define amdgpu_kernel void @v_test_umin_ule_v3i32(ptr addrspace(1) %out, ptr addr ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v6, 4, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v6 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_mov_b32_e32 v2, s5 ; VI-NEXT: v_add_u32_e32 v3, vcc, s4, v6 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v4, vcc, 0, v2, vcc ; VI-NEXT: flat_load_dwordx3 v[0:2], v[0:1] ; VI-NEXT: flat_load_dwordx3 v[3:5], v[3:4] @@ -1838,12 +1934,15 @@ define amdgpu_kernel void @v_test_umin_ule_v3i16(ptr addrspace(1) %out, ptr addr ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: v_add_i32_e32 v2, vcc, s4, v4 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; CI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; CI-NEXT: flat_load_dwordx2 v[2:3], v[2:3] @@ -1874,12 +1973,15 @@ define amdgpu_kernel void @v_test_umin_ule_v3i16(ptr addrspace(1) %out, ptr addr ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc ; VI-NEXT: flat_load_dwordx2 v[0:1], v[0:1] ; VI-NEXT: flat_load_dwordx2 v[2:3], v[2:3] @@ -1976,6 +2078,9 @@ define amdgpu_kernel void @s_test_umin_ule_i32(ptr addrspace(1) %out, i32 %a, i3 ; CI-LABEL: s_test_umin_ule_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_u32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -1987,6 +2092,9 @@ define amdgpu_kernel void @s_test_umin_ule_i32(ptr addrspace(1) %out, i32 %a, i3 ; VI-LABEL: s_test_umin_ule_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_u32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -2059,10 +2167,13 @@ define amdgpu_kernel void @v_test_umin_ult_i32(ptr addrspace(1) %out, ptr addrsp ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v4 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: v_add_i32_e32 v2, vcc, s4, v4 ; CI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -2081,10 +2192,13 @@ define amdgpu_kernel void @v_test_umin_ult_i32(ptr addrspace(1) %out, ptr addrsp ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -2188,6 +2302,9 @@ define amdgpu_kernel void @v_test_umin_ult_i8(ptr addrspace(1) %out, ptr addrspa ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v2, s3 ; CI-NEXT: v_add_i32_e32 v1, vcc, s2, v0 @@ -2209,6 +2326,9 @@ define amdgpu_kernel void @v_test_umin_ult_i8(ptr addrspace(1) %out, ptr addrspa ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v2, s3 ; VI-NEXT: v_add_u32_e32 v1, vcc, s2, v0 @@ -2294,6 +2414,9 @@ define amdgpu_kernel void @s_test_umin_ult_i32(ptr addrspace(1) %out, i32 %a, i3 ; CI-LABEL: s_test_umin_ult_i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_u32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -2305,6 +2428,9 @@ define amdgpu_kernel void @s_test_umin_ult_i32(ptr addrspace(1) %out, i32 %a, i3 ; VI-LABEL: s_test_umin_ult_i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_u32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -2386,6 +2512,9 @@ define amdgpu_kernel void @v_test_umin_ult_i32_multi_use(ptr addrspace(1) %out0, ; CI-LABEL: v_test_umin_ult_i32_multi_use: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_load_dword s4, s[4:5], 0x0 ; CI-NEXT: s_load_dword s5, s[6:7], 0x0 @@ -2407,6 +2536,9 @@ define amdgpu_kernel void @v_test_umin_ult_i32_multi_use(ptr addrspace(1) %out0, ; VI-LABEL: v_test_umin_ult_i32_multi_use: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_load_dword s4, s[4:5], 0x0 ; VI-NEXT: s_load_dword s5, s[6:7], 0x0 @@ -2534,6 +2666,9 @@ define amdgpu_kernel void @v_test_umin_ult_i16_multi_use(ptr addrspace(1) %out0, ; CI-LABEL: v_test_umin_ult_i16_multi_use: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s4 ; CI-NEXT: v_mov_b32_e32 v1, s5 @@ -2556,6 +2691,9 @@ define amdgpu_kernel void @v_test_umin_ult_i16_multi_use(ptr addrspace(1) %out0, ; VI-LABEL: v_test_umin_ult_i16_multi_use: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s4 ; VI-NEXT: v_mov_b32_e32 v1, s5 @@ -2646,6 +2784,9 @@ define amdgpu_kernel void @s_test_umin_ult_v1i32(ptr addrspace(1) %out, <1 x i32 ; CI-LABEL: s_test_umin_ult_v1i32: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_min_u32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 @@ -2657,6 +2798,9 @@ define amdgpu_kernel void @s_test_umin_ult_v1i32(ptr addrspace(1) %out, <1 x i32 ; VI-LABEL: s_test_umin_ult_v1i32: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_min_u32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 @@ -2726,6 +2870,9 @@ define amdgpu_kernel void @s_test_umin_ult_v8i32(ptr addrspace(1) %out, <8 x i32 ; ; CI-LABEL: s_test_umin_ult_v8i32: ; CI: ; %bb.0: +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_load_dwordx16 s[12:27], s[8:9], 0x8 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; CI-NEXT: s_waitcnt lgkmcnt(0) @@ -2757,6 +2904,9 @@ define amdgpu_kernel void @s_test_umin_ult_v8i32(ptr addrspace(1) %out, <8 x i32 ; ; VI-LABEL: s_test_umin_ult_v8i32: ; VI: ; %bb.0: +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_load_dwordx16 s[12:27], s[8:9], 0x20 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; VI-NEXT: s_waitcnt lgkmcnt(0) @@ -2921,6 +3071,9 @@ define amdgpu_kernel void @s_test_umin_ult_v8i16(ptr addrspace(1) %out, <8 x i16 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x4 ; CI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_lshr_b32 s10, s0, 16 ; CI-NEXT: s_and_b32 s0, s0, 0xffff @@ -2967,6 +3120,9 @@ define amdgpu_kernel void @s_test_umin_ult_v8i16(ptr addrspace(1) %out, <8 x i16 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x10 ; VI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_lshr_b32 s10, s3, 16 ; VI-NEXT: s_and_b32 s3, s3, 0xffff @@ -3088,11 +3244,14 @@ define amdgpu_kernel void @simplify_demanded_bits_test_umin_ult_i16(ptr addrspac ; CI-NEXT: s_load_dword s2, s[8:9], 0xa ; CI-NEXT: s_load_dword s3, s[8:9], 0x13 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_and_b32 s2, s2, 0xffff ; CI-NEXT: s_and_b32 s3, s3, 0xffff ; CI-NEXT: s_min_u32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: flat_store_dword v[0:1], v2 @@ -3103,11 +3262,14 @@ define amdgpu_kernel void @simplify_demanded_bits_test_umin_ult_i16(ptr addrspac ; VI-NEXT: s_load_dword s2, s[8:9], 0x28 ; VI-NEXT: s_load_dword s3, s[8:9], 0x4c ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_and_b32 s2, s2, 0xffff ; VI-NEXT: s_and_b32 s3, s3, 0xffff ; VI-NEXT: s_min_u32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: flat_store_dword v[0:1], v2 @@ -3195,11 +3357,14 @@ define amdgpu_kernel void @simplify_demanded_bits_test_min_slt_i16(ptr addrspace ; CI-NEXT: s_load_dword s2, s[8:9], 0xa ; CI-NEXT: s_load_dword s3, s[8:9], 0x13 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_sext_i32_i16 s2, s2 ; CI-NEXT: s_sext_i32_i16 s3, s3 ; CI-NEXT: s_min_i32 s2, s2, s3 ; CI-NEXT: v_mov_b32_e32 v0, s0 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_mov_b32_e32 v1, s1 ; CI-NEXT: v_mov_b32_e32 v2, s2 ; CI-NEXT: flat_store_dword v[0:1], v2 @@ -3210,11 +3375,14 @@ define amdgpu_kernel void @simplify_demanded_bits_test_min_slt_i16(ptr addrspace ; VI-NEXT: s_load_dword s2, s[8:9], 0x28 ; VI-NEXT: s_load_dword s3, s[8:9], 0x4c ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_sext_i32_i16 s2, s2 ; VI-NEXT: s_sext_i32_i16 s3, s3 ; VI-NEXT: s_min_i32 s2, s2, s3 ; VI-NEXT: v_mov_b32_e32 v0, s0 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v1, s1 ; VI-NEXT: v_mov_b32_e32 v2, s2 ; VI-NEXT: flat_store_dword v[0:1], v2 @@ -3309,6 +3477,9 @@ define amdgpu_kernel void @s_test_imin_sle_i16(ptr addrspace(1) %out, i16 %a, i1 ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_sext_i32_i16 s3, s2 ; CI-NEXT: s_ashr_i32 s2, s2, 16 @@ -3323,6 +3494,9 @@ define amdgpu_kernel void @s_test_imin_sle_i16(ptr addrspace(1) %out, i16 %a, i1 ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_sext_i32_i16 s3, s2 ; VI-NEXT: s_ashr_i32 s2, s2, 16 @@ -3403,6 +3577,9 @@ define amdgpu_kernel void @test_umin_ult_i64(ptr addrspace(1) %out, i64 %a, i64 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s4 @@ -3421,6 +3598,9 @@ define amdgpu_kernel void @test_umin_ult_i64(ptr addrspace(1) %out, i64 %a, i64 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s4 @@ -3510,6 +3690,9 @@ define amdgpu_kernel void @test_umin_ule_i64(ptr addrspace(1) %out, i64 %a, i64 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s4 @@ -3528,6 +3711,9 @@ define amdgpu_kernel void @test_umin_ule_i64(ptr addrspace(1) %out, i64 %a, i64 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s4 @@ -3617,6 +3803,9 @@ define amdgpu_kernel void @test_imin_slt_i64(ptr addrspace(1) %out, i64 %a, i64 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s4 @@ -3635,6 +3824,9 @@ define amdgpu_kernel void @test_imin_slt_i64(ptr addrspace(1) %out, i64 %a, i64 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s4 @@ -3724,6 +3916,9 @@ define amdgpu_kernel void @test_imin_sle_i64(ptr addrspace(1) %out, i64 %a, i64 ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v0, s0 ; CI-NEXT: v_mov_b32_e32 v1, s4 @@ -3742,6 +3937,9 @@ define amdgpu_kernel void @test_imin_sle_i64(ptr addrspace(1) %out, i64 %a, i64 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v0, s0 ; VI-NEXT: v_mov_b32_e32 v1, s4 @@ -3855,9 +4053,12 @@ define amdgpu_kernel void @v_test_imin_sle_v2i16(ptr addrspace(1) %out, ptr addr ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: flat_load_dword v4, v[0:1] @@ -3886,10 +4087,13 @@ define amdgpu_kernel void @v_test_imin_sle_v2i16(ptr addrspace(1) %out, ptr addr ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc @@ -4005,9 +4209,12 @@ define amdgpu_kernel void @v_test_imin_ule_v2i16(ptr addrspace(1) %out, ptr addr ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x4 ; CI-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v2 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc ; CI-NEXT: v_mov_b32_e32 v3, s5 ; CI-NEXT: flat_load_dword v4, v[0:1] @@ -4035,10 +4242,13 @@ define amdgpu_kernel void @v_test_imin_ule_v2i16(ptr addrspace(1) %out, ptr addr ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10 ; VI-NEXT: v_lshlrev_b32_e32 v4, 2, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v4 ; VI-NEXT: v_addc_u32_e32 v1, vcc, 0, v1, vcc +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: v_mov_b32_e32 v3, s5 ; VI-NEXT: v_add_u32_e32 v2, vcc, s4, v4 ; VI-NEXT: v_addc_u32_e32 v3, vcc, 0, v3, vcc diff --git a/llvm/test/CodeGen/AMDGPU/pack.v2f16.ll b/llvm/test/CodeGen/AMDGPU/pack.v2f16.ll index 337320b9eeea1..b1ce5a3423f20 100644 --- a/llvm/test/CodeGen/AMDGPU/pack.v2f16.ll +++ b/llvm/test/CodeGen/AMDGPU/pack.v2f16.ll @@ -180,6 +180,9 @@ define amdgpu_kernel void @v_pack_v2f16(ptr addrspace(1) %in0, ptr addrspace(1) ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -260,6 +263,9 @@ define amdgpu_kernel void @v_pack_v2f16_user(ptr addrspace(1) %in0, ptr addrspac ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -341,6 +347,9 @@ define amdgpu_kernel void @v_pack_v2f16_imm_lo(ptr addrspace(1) %in1) #0 { ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -403,6 +412,9 @@ define amdgpu_kernel void @v_pack_v2f16_inline_imm_lo(ptr addrspace(1) %in1) #0 ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -465,6 +477,9 @@ define amdgpu_kernel void @v_pack_v2f16_imm_hi(ptr addrspace(1) %in0) #0 { ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -527,6 +542,9 @@ define amdgpu_kernel void @v_pack_v2f16_inline_f16imm_hi(ptr addrspace(1) %in0) ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -588,6 +606,9 @@ define amdgpu_kernel void @v_pack_v2f16_inline_imm_hi(ptr addrspace(1) %in0) #0 ; GFX8: ; %bb.0: ; GFX8-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX8-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX8-NEXT: s_add_i32 s12, s12, s17 +; GFX8-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX8-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX8-NEXT: s_waitcnt lgkmcnt(0) ; GFX8-NEXT: v_mov_b32_e32 v1, s1 ; GFX8-NEXT: v_add_u32_e32 v0, vcc, s0, v0 diff --git a/llvm/test/CodeGen/AMDGPU/pack.v2i16.ll b/llvm/test/CodeGen/AMDGPU/pack.v2i16.ll index bc1710686a087..5803821a1d2c0 100644 --- a/llvm/test/CodeGen/AMDGPU/pack.v2i16.ll +++ b/llvm/test/CodeGen/AMDGPU/pack.v2i16.ll @@ -176,6 +176,9 @@ define amdgpu_kernel void @v_pack_v2i16(ptr addrspace(1) %in0, ptr addrspace(1) ; GFX803: ; %bb.0: ; GFX803-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX803-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX803-NEXT: s_add_i32 s12, s12, s17 +; GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX803-NEXT: s_waitcnt lgkmcnt(0) ; GFX803-NEXT: v_mov_b32_e32 v1, s1 ; GFX803-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -254,6 +257,9 @@ define amdgpu_kernel void @v_pack_v2i16_user(ptr addrspace(1) %in0, ptr addrspac ; GFX803: ; %bb.0: ; GFX803-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX803-NEXT: v_lshlrev_b32_e32 v2, 2, v0 +; GFX803-NEXT: s_add_i32 s12, s12, s17 +; GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX803-NEXT: s_waitcnt lgkmcnt(0) ; GFX803-NEXT: v_mov_b32_e32 v1, s1 ; GFX803-NEXT: v_add_u32_e32 v0, vcc, s0, v2 @@ -333,6 +339,9 @@ define amdgpu_kernel void @v_pack_v2i16_imm_lo(ptr addrspace(1) %in1) #0 { ; GFX803: ; %bb.0: ; GFX803-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX803-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX803-NEXT: s_add_i32 s12, s12, s17 +; GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX803-NEXT: s_waitcnt lgkmcnt(0) ; GFX803-NEXT: v_mov_b32_e32 v1, s1 ; GFX803-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -393,6 +402,9 @@ define amdgpu_kernel void @v_pack_v2i16_inline_imm_lo(ptr addrspace(1) %in1) #0 ; GFX803: ; %bb.0: ; GFX803-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX803-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX803-NEXT: s_add_i32 s12, s12, s17 +; GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX803-NEXT: s_waitcnt lgkmcnt(0) ; GFX803-NEXT: v_mov_b32_e32 v1, s1 ; GFX803-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -454,6 +466,9 @@ define amdgpu_kernel void @v_pack_v2i16_imm_hi(ptr addrspace(1) %in0) #0 { ; GFX803: ; %bb.0: ; GFX803-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX803-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX803-NEXT: s_add_i32 s12, s12, s17 +; GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX803-NEXT: s_waitcnt lgkmcnt(0) ; GFX803-NEXT: v_mov_b32_e32 v1, s1 ; GFX803-NEXT: v_add_u32_e32 v0, vcc, s0, v0 @@ -514,6 +529,9 @@ define amdgpu_kernel void @v_pack_v2i16_inline_imm_hi(ptr addrspace(1) %in0) #0 ; GFX803: ; %bb.0: ; GFX803-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GFX803-NEXT: v_lshlrev_b32_e32 v0, 2, v0 +; GFX803-NEXT: s_add_i32 s12, s12, s17 +; GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GFX803-NEXT: s_waitcnt lgkmcnt(0) ; GFX803-NEXT: v_mov_b32_e32 v1, s1 ; GFX803-NEXT: v_add_u32_e32 v0, vcc, s0, v0 diff --git a/llvm/test/CodeGen/AMDGPU/pal-simple-indirect-call.ll b/llvm/test/CodeGen/AMDGPU/pal-simple-indirect-call.ll index 2e9f09ad41813..7c9ecc892478c 100644 --- a/llvm/test/CodeGen/AMDGPU/pal-simple-indirect-call.ll +++ b/llvm/test/CodeGen/AMDGPU/pal-simple-indirect-call.ll @@ -1,6 +1,5 @@ ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals ; Check that no attributes are added to graphics functions -; RUN: opt -S -mtriple=amdgcn-amd-amdpal -amdgpu-annotate-kernel-features %s | FileCheck -check-prefixes=AKF_GCN %s ; RUN: opt -S -mtriple=amdgcn-amd-amdpal -passes=amdgpu-attributor %s | FileCheck -check-prefixes=ATTRIBUTOR_GCN %s ; Check that it doesn't crash @@ -12,12 +11,6 @@ target datalayout = "A5" define amdgpu_cs void @test_simple_indirect_call() { -; AKF_GCN-LABEL: define {{[^@]+}}@test_simple_indirect_call() { -; AKF_GCN-NEXT: [[PC:%.*]] = call i64 @llvm.amdgcn.s.getpc() -; AKF_GCN-NEXT: [[FUN:%.*]] = inttoptr i64 [[PC]] to ptr -; AKF_GCN-NEXT: call amdgpu_gfx void [[FUN]]() -; AKF_GCN-NEXT: ret void -; ; ATTRIBUTOR_GCN-LABEL: define {{[^@]+}}@test_simple_indirect_call ; ATTRIBUTOR_GCN-SAME: () #[[ATTR0:[0-9]+]] { ; ATTRIBUTOR_GCN-NEXT: [[PC:%.*]] = call i64 @llvm.amdgcn.s.getpc() @@ -68,7 +61,6 @@ declare i64 @llvm.amdgcn.s.getpc() #0 attributes #0 = { nounwind readnone speculatable willreturn } ;. -; AKF_GCN: attributes #[[ATTR0:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } ;. ; ATTRIBUTOR_GCN: attributes #[[ATTR0]] = { "uniform-work-group-size"="false" } ; ATTRIBUTOR_GCN: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } diff --git a/llvm/test/CodeGen/AMDGPU/partial-regcopy-and-spill-missed-at-regalloc.ll b/llvm/test/CodeGen/AMDGPU/partial-regcopy-and-spill-missed-at-regalloc.ll index 1a34fa3bbbf4d..24677b60be6c2 100644 --- a/llvm/test/CodeGen/AMDGPU/partial-regcopy-and-spill-missed-at-regalloc.ll +++ b/llvm/test/CodeGen/AMDGPU/partial-regcopy-and-spill-missed-at-regalloc.ll @@ -10,32 +10,33 @@ define amdgpu_kernel void @partial_copy(<4 x i32> %arg) #0 { ; REGALLOC-GFX908: bb.0 (%ir-block.0): ; REGALLOC-GFX908-NEXT: liveins: $sgpr4_sgpr5 ; REGALLOC-GFX908-NEXT: {{ $}} - ; REGALLOC-GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 2162697 /* reguse:AGPR_32 */, undef %5:agpr_32 - ; REGALLOC-GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6094858 /* regdef:VReg_128 */, def %6 - ; REGALLOC-GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 3538954 /* regdef:VReg_64 */, def %7 - ; REGALLOC-GFX908-NEXT: GLOBAL_STORE_DWORDX4 undef %14:vreg_64, %6, 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) + ; REGALLOC-GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 2162697 /* reguse:AGPR_32 */, undef %6:agpr_32 + ; REGALLOC-GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6094858 /* regdef:VReg_128 */, def %7 + ; REGALLOC-GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 3538954 /* regdef:VReg_64 */, def %8 + + ; REGALLOC-GFX908-NEXT: GLOBAL_STORE_DWORDX4 undef %15:vreg_64, %7, 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) ; REGALLOC-GFX908-NEXT: renamable $sgpr0_sgpr1_sgpr2_sgpr3 = S_LOAD_DWORDX4_IMM killed renamable $sgpr4_sgpr5, 0, 0 :: (dereferenceable invariant load (s128) from %ir.arg.kernarg.offset1, addrspace 4) ; REGALLOC-GFX908-NEXT: [[COPY:%[0-9]+]]:areg_128 = COPY killed renamable $sgpr0_sgpr1_sgpr2_sgpr3 ; REGALLOC-GFX908-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 1, implicit $exec ; REGALLOC-GFX908-NEXT: [[V_MOV_B32_e32_1:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 2, implicit $exec ; REGALLOC-GFX908-NEXT: [[V_MFMA_I32_4X4X4I8_e64_:%[0-9]+]]:areg_128 = V_MFMA_I32_4X4X4I8_e64 [[V_MOV_B32_e32_]], [[V_MOV_B32_e32_1]], [[COPY]], 0, 0, 0, implicit $mode, implicit $exec - ; REGALLOC-GFX908-NEXT: GLOBAL_STORE_DWORDX2 undef %16:vreg_64, %7, 0, 0, implicit $exec :: (volatile store (s64) into `ptr addrspace(1) poison`, addrspace 1) - ; REGALLOC-GFX908-NEXT: [[COPY1:%[0-9]+]]:vreg_128 = COPY [[V_MFMA_I32_4X4X4I8_e64_]] - ; REGALLOC-GFX908-NEXT: GLOBAL_STORE_DWORDX4 undef %18:vreg_64, [[COPY1]], 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) + ; REGALLOC-GFX908-NEXT: GLOBAL_STORE_DWORDX2 undef %17:vreg_64, %8, 0, 0, implicit $exec :: (volatile store (s64) into `ptr addrspace(1) poison`, addrspace 1) + ; REGALLOC-GFX908-NEXT: [[COPY3:%[0-9]+]]:vreg_128 = COPY [[V_MFMA_I32_4X4X4I8_e64_]] + ; REGALLOC-GFX908-NEXT: GLOBAL_STORE_DWORDX4 undef %19:vreg_64, [[COPY3]], 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) ; REGALLOC-GFX908-NEXT: S_ENDPGM 0 ; ; PEI-GFX908-LABEL: name: partial_copy ; PEI-GFX908: bb.0 (%ir-block.0): - ; PEI-GFX908-NEXT: liveins: $agpr4, $sgpr4_sgpr5, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr7 + ; PEI-GFX908-NEXT: liveins: $agpr4, $sgpr4_sgpr5, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr9 ; PEI-GFX908-NEXT: {{ $}} - ; PEI-GFX908-NEXT: $sgpr8_sgpr9_sgpr10_sgpr11 = COPY killed $sgpr0_sgpr1_sgpr2_sgpr3 - ; PEI-GFX908-NEXT: $sgpr8 = S_ADD_U32 $sgpr8, $sgpr7, implicit-def $scc, implicit-def $sgpr8_sgpr9_sgpr10_sgpr11 - ; PEI-GFX908-NEXT: $sgpr9 = S_ADDC_U32 $sgpr9, 0, implicit-def dead $scc, implicit $scc, implicit-def $sgpr8_sgpr9_sgpr10_sgpr11 + ; PEI-GFX908-NEXT: $sgpr12_sgpr13_sgpr14_sgpr15 = COPY killed $sgpr0_sgpr1_sgpr2_sgpr3 + ; PEI-GFX908-NEXT: $sgpr12 = S_ADD_U32 $sgpr12, $sgpr9, implicit-def $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15 + ; PEI-GFX908-NEXT: $sgpr13 = S_ADDC_U32 $sgpr13, 0, implicit-def dead $scc, implicit $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15 ; PEI-GFX908-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 2162697 /* reguse:AGPR_32 */, undef renamable $agpr0 ; PEI-GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6094858 /* regdef:VReg_128 */, def renamable $vgpr0_vgpr1_vgpr2_vgpr3 ; PEI-GFX908-NEXT: renamable $agpr0_agpr1_agpr2_agpr3 = COPY killed renamable $vgpr0_vgpr1_vgpr2_vgpr3, implicit $exec ; PEI-GFX908-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 3538954 /* regdef:VReg_64 */, def renamable $vgpr0_vgpr1 - ; PEI-GFX908-NEXT: BUFFER_STORE_DWORD_OFFSET killed $vgpr0, $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1, implicit $vgpr0_vgpr1 :: (store (s32) into %stack.0, addrspace 5) + ; PEI-GFX908-NEXT: BUFFER_STORE_DWORD_OFFSET killed $vgpr0, $sgpr12_sgpr13_sgpr14_sgpr15, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1, implicit $vgpr0_vgpr1 :: (store (s32) into %stack.0, addrspace 5) ; PEI-GFX908-NEXT: $agpr4 = V_ACCVGPR_WRITE_B32_e64 killed $vgpr1, implicit $exec, implicit killed $vgpr0_vgpr1 ; PEI-GFX908-NEXT: renamable $vgpr0_vgpr1_vgpr2_vgpr3 = COPY killed renamable $agpr0_agpr1_agpr2_agpr3, implicit $exec ; PEI-GFX908-NEXT: GLOBAL_STORE_DWORDX4 undef renamable $vgpr0_vgpr1, killed renamable $vgpr0_vgpr1_vgpr2_vgpr3, 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) @@ -44,7 +45,7 @@ define amdgpu_kernel void @partial_copy(<4 x i32> %arg) #0 { ; PEI-GFX908-NEXT: renamable $vgpr0 = V_MOV_B32_e32 1, implicit $exec ; PEI-GFX908-NEXT: renamable $vgpr1 = V_MOV_B32_e32 2, implicit $exec ; PEI-GFX908-NEXT: renamable $agpr0_agpr1_agpr2_agpr3 = V_MFMA_I32_4X4X4I8_e64 killed $vgpr0, killed $vgpr1, killed $agpr0_agpr1_agpr2_agpr3, 0, 0, 0, implicit $mode, implicit $exec - ; PEI-GFX908-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1 :: (load (s32) from %stack.0, addrspace 5) + ; PEI-GFX908-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr12_sgpr13_sgpr14_sgpr15, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1 :: (load (s32) from %stack.0, addrspace 5) ; PEI-GFX908-NEXT: $vgpr1 = V_ACCVGPR_READ_B32_e64 $agpr4, implicit $exec, implicit $vgpr0_vgpr1 ; PEI-GFX908-NEXT: GLOBAL_STORE_DWORDX2 undef renamable $vgpr0_vgpr1, killed renamable $vgpr0_vgpr1, 0, 0, implicit $exec :: (volatile store (s64) into `ptr addrspace(1) poison`, addrspace 1) ; PEI-GFX908-NEXT: renamable $vgpr0_vgpr1_vgpr2_vgpr3 = COPY killed renamable $agpr0_agpr1_agpr2_agpr3, implicit $exec @@ -55,31 +56,31 @@ define amdgpu_kernel void @partial_copy(<4 x i32> %arg) #0 { ; REGALLOC-GFX90A: bb.0 (%ir-block.0): ; REGALLOC-GFX90A-NEXT: liveins: $sgpr4_sgpr5 ; REGALLOC-GFX90A-NEXT: {{ $}} - ; REGALLOC-GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 2162697 /* reguse:AGPR_32 */, undef %5:agpr_32 - ; REGALLOC-GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6422538 /* regdef:VReg_128_Align2 */, def %6 - ; REGALLOC-GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 3866634 /* regdef:VReg_64_Align2 */, def %7 - ; REGALLOC-GFX90A-NEXT: GLOBAL_STORE_DWORDX4 undef %14:vreg_64_align2, %6, 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) + ; REGALLOC-GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 2162697 /* reguse:AGPR_32 */, undef %6:agpr_32 + ; REGALLOC-GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6422538 /* regdef:VReg_128_Align2 */, def %7 + ; REGALLOC-GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 3866634 /* regdef:VReg_64_Align2 */, def %8 + ; REGALLOC-GFX90A-NEXT: GLOBAL_STORE_DWORDX4 undef %15:vreg_64_align2, %7, 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) ; REGALLOC-GFX90A-NEXT: renamable $sgpr0_sgpr1_sgpr2_sgpr3 = S_LOAD_DWORDX4_IMM killed renamable $sgpr4_sgpr5, 0, 0 :: (dereferenceable invariant load (s128) from %ir.arg.kernarg.offset1, addrspace 4) ; REGALLOC-GFX90A-NEXT: [[COPY:%[0-9]+]]:areg_128_align2 = COPY killed renamable $sgpr0_sgpr1_sgpr2_sgpr3 ; REGALLOC-GFX90A-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 1, implicit $exec ; REGALLOC-GFX90A-NEXT: [[V_MOV_B32_e32_1:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 2, implicit $exec ; REGALLOC-GFX90A-NEXT: [[V_MFMA_I32_4X4X4I8_e64_:%[0-9]+]]:areg_128_align2 = V_MFMA_I32_4X4X4I8_e64 [[V_MOV_B32_e32_]], [[V_MOV_B32_e32_1]], [[COPY]], 0, 0, 0, implicit $mode, implicit $exec - ; REGALLOC-GFX90A-NEXT: GLOBAL_STORE_DWORDX2 undef %16:vreg_64_align2, %7, 0, 0, implicit $exec :: (volatile store (s64) into `ptr addrspace(1) poison`, addrspace 1) - ; REGALLOC-GFX90A-NEXT: GLOBAL_STORE_DWORDX4 undef %18:vreg_64_align2, [[V_MFMA_I32_4X4X4I8_e64_]], 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) + ; REGALLOC-GFX90A-NEXT: GLOBAL_STORE_DWORDX2 undef %17:vreg_64_align2, %8, 0, 0, implicit $exec :: (volatile store (s64) into `ptr addrspace(1) poison`, addrspace 1) + ; REGALLOC-GFX90A-NEXT: GLOBAL_STORE_DWORDX4 undef %19:vreg_64_align2, [[V_MFMA_I32_4X4X4I8_e64_]], 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) ; REGALLOC-GFX90A-NEXT: S_ENDPGM 0 ; ; PEI-GFX90A-LABEL: name: partial_copy ; PEI-GFX90A: bb.0 (%ir-block.0): - ; PEI-GFX90A-NEXT: liveins: $agpr4, $sgpr4_sgpr5, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr7 + ; PEI-GFX90A-NEXT: liveins: $agpr4, $sgpr4_sgpr5, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr9 ; PEI-GFX90A-NEXT: {{ $}} - ; PEI-GFX90A-NEXT: $sgpr8_sgpr9_sgpr10_sgpr11 = COPY killed $sgpr0_sgpr1_sgpr2_sgpr3 - ; PEI-GFX90A-NEXT: $sgpr8 = S_ADD_U32 $sgpr8, $sgpr7, implicit-def $scc, implicit-def $sgpr8_sgpr9_sgpr10_sgpr11 - ; PEI-GFX90A-NEXT: $sgpr9 = S_ADDC_U32 $sgpr9, 0, implicit-def dead $scc, implicit $scc, implicit-def $sgpr8_sgpr9_sgpr10_sgpr11 + ; PEI-GFX90A-NEXT: $sgpr12_sgpr13_sgpr14_sgpr15 = COPY killed $sgpr0_sgpr1_sgpr2_sgpr3 + ; PEI-GFX90A-NEXT: $sgpr12 = S_ADD_U32 $sgpr12, $sgpr9, implicit-def $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15 + ; PEI-GFX90A-NEXT: $sgpr13 = S_ADDC_U32 $sgpr13, 0, implicit-def dead $scc, implicit $scc, implicit-def $sgpr12_sgpr13_sgpr14_sgpr15 ; PEI-GFX90A-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 2162697 /* reguse:AGPR_32 */, undef renamable $agpr0 ; PEI-GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 6422538 /* regdef:VReg_128_Align2 */, def renamable $vgpr0_vgpr1_vgpr2_vgpr3 ; PEI-GFX90A-NEXT: renamable $agpr0_agpr1_agpr2_agpr3 = COPY killed renamable $vgpr0_vgpr1_vgpr2_vgpr3, implicit $exec ; PEI-GFX90A-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 3866634 /* regdef:VReg_64_Align2 */, def renamable $vgpr0_vgpr1 - ; PEI-GFX90A-NEXT: BUFFER_STORE_DWORD_OFFSET killed $vgpr0, $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1, implicit $vgpr0_vgpr1 :: (store (s32) into %stack.0, addrspace 5) + ; PEI-GFX90A-NEXT: BUFFER_STORE_DWORD_OFFSET killed $vgpr0, $sgpr12_sgpr13_sgpr14_sgpr15, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1, implicit $vgpr0_vgpr1 :: (store (s32) into %stack.0, addrspace 5) ; PEI-GFX90A-NEXT: $agpr4 = V_ACCVGPR_WRITE_B32_e64 killed $vgpr1, implicit $exec, implicit killed $vgpr0_vgpr1 ; PEI-GFX90A-NEXT: GLOBAL_STORE_DWORDX4 undef renamable $vgpr0_vgpr1, killed renamable $agpr0_agpr1_agpr2_agpr3, 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) ; PEI-GFX90A-NEXT: renamable $sgpr0_sgpr1_sgpr2_sgpr3 = S_LOAD_DWORDX4_IMM killed renamable $sgpr4_sgpr5, 0, 0 :: (dereferenceable invariant load (s128) from %ir.arg.kernarg.offset1, addrspace 4) @@ -87,7 +88,7 @@ define amdgpu_kernel void @partial_copy(<4 x i32> %arg) #0 { ; PEI-GFX90A-NEXT: renamable $vgpr0 = V_MOV_B32_e32 1, implicit $exec ; PEI-GFX90A-NEXT: renamable $vgpr1 = V_MOV_B32_e32 2, implicit $exec ; PEI-GFX90A-NEXT: renamable $agpr0_agpr1_agpr2_agpr3 = V_MFMA_I32_4X4X4I8_e64 killed $vgpr0, killed $vgpr1, killed $agpr0_agpr1_agpr2_agpr3, 0, 0, 0, implicit $mode, implicit $exec - ; PEI-GFX90A-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1 :: (load (s32) from %stack.0, addrspace 5) + ; PEI-GFX90A-NEXT: $vgpr0 = BUFFER_LOAD_DWORD_OFFSET $sgpr12_sgpr13_sgpr14_sgpr15, 0, 0, 0, 0, implicit $exec, implicit-def $vgpr0_vgpr1 :: (load (s32) from %stack.0, addrspace 5) ; PEI-GFX90A-NEXT: $vgpr1 = V_ACCVGPR_READ_B32_e64 $agpr4, implicit $exec, implicit $vgpr0_vgpr1 ; PEI-GFX90A-NEXT: GLOBAL_STORE_DWORDX2 undef renamable $vgpr0_vgpr1, killed renamable $vgpr0_vgpr1, 0, 0, implicit $exec :: (volatile store (s64) into `ptr addrspace(1) poison`, addrspace 1) ; PEI-GFX90A-NEXT: GLOBAL_STORE_DWORDX4 undef renamable $vgpr0_vgpr1, killed renamable $agpr0_agpr1_agpr2_agpr3, 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) diff --git a/llvm/test/CodeGen/AMDGPU/preload-implicit-kernargs.ll b/llvm/test/CodeGen/AMDGPU/preload-implicit-kernargs.ll index 00507c1eafd6e..c26f0926d86b2 100644 --- a/llvm/test/CodeGen/AMDGPU/preload-implicit-kernargs.ll +++ b/llvm/test/CodeGen/AMDGPU/preload-implicit-kernargs.ll @@ -19,16 +19,16 @@ define amdgpu_kernel void @preload_block_count_x(ptr addrspace(1) inreg %out) #0 ; ; GFX90a-LABEL: preload_block_count_x: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB0_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB0_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %load = load i32, ptr addrspace(4) %imp_arg_ptr @@ -54,17 +54,16 @@ define amdgpu_kernel void @preload_unused_arg_block_count_x(ptr addrspace(1) inr ; ; GFX90a-LABEL: preload_unused_arg_block_count_x: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x10 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB1_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB1_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s10 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s12 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %load = load i32, ptr addrspace(4) %imp_arg_ptr @@ -90,7 +89,7 @@ define amdgpu_kernel void @no_free_sgprs_block_count_x(ptr addrspace(1) inreg %o ; ; GFX90a-LABEL: no_free_sgprs_block_count_x: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx4 s[12:15], s[8:9], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[14:15], s[8:9], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB2_0 ; GFX90a-NEXT: .p2align 8 @@ -100,7 +99,7 @@ define amdgpu_kernel void @no_free_sgprs_block_count_x(ptr addrspace(1) inreg %o ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[12:13] +; GFX90a-NEXT: global_store_dword v0, v1, s[14:15] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %load = load i32, ptr addrspace(4) %imp_arg_ptr @@ -181,7 +180,7 @@ define amdgpu_kernel void @incorrect_type_i64_block_count_x(ptr addrspace(1) inr ; ; GFX90a-LABEL: incorrect_type_i64_block_count_x: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB5_0 ; GFX90a-NEXT: .p2align 8 @@ -191,7 +190,7 @@ define amdgpu_kernel void @incorrect_type_i64_block_count_x(ptr addrspace(1) inr ; GFX90a-NEXT: v_mov_b32_e32 v2, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_pk_mov_b32 v[0:1], s[0:1], s[0:1] op_sel:[0,1] -; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[6:7] +; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %load = load i64, ptr addrspace(4) %imp_arg_ptr @@ -217,7 +216,7 @@ define amdgpu_kernel void @incorrect_type_i16_block_count_x(ptr addrspace(1) inr ; ; GFX90a-LABEL: incorrect_type_i16_block_count_x: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB6_0 ; GFX90a-NEXT: .p2align 8 @@ -227,7 +226,7 @@ define amdgpu_kernel void @incorrect_type_i16_block_count_x(ptr addrspace(1) inr ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %load = load i16, ptr addrspace(4) %imp_arg_ptr @@ -252,16 +251,15 @@ define amdgpu_kernel void @preload_block_count_y(ptr addrspace(1) inreg %out) #0 ; ; GFX90a-LABEL: preload_block_count_y: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB7_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB7_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s11 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 4 @@ -289,7 +287,7 @@ define amdgpu_kernel void @random_incorrect_offset(ptr addrspace(1) inreg %out) ; ; GFX90a-LABEL: random_incorrect_offset: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB8_0 ; GFX90a-NEXT: .p2align 8 @@ -300,7 +298,7 @@ define amdgpu_kernel void @random_incorrect_offset(ptr addrspace(1) inreg %out) ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 2 @@ -327,17 +325,16 @@ define amdgpu_kernel void @preload_block_count_z(ptr addrspace(1) inreg %out) #0 ; ; GFX90a-LABEL: preload_block_count_z: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x10 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB9_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB9_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s10 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s12 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 8 @@ -366,19 +363,18 @@ define amdgpu_kernel void @preload_block_count_x_imparg_align_ptr_i8(ptr addrspa ; ; GFX90a-LABEL: preload_block_count_x_imparg_align_ptr_i8: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x10 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB10_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB10_0: -; GFX90a-NEXT: s_and_b32 s0, s8, 0xff -; GFX90a-NEXT: s_add_i32 s0, s10, s0 +; GFX90a-NEXT: s_and_b32 s0, s10, 0xff +; GFX90a-NEXT: s_add_i32 s0, s12, s0 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %load = load i32, ptr addrspace(4) %imp_arg_ptr @@ -408,19 +404,18 @@ define amdgpu_kernel void @preload_block_count_xyz(ptr addrspace(1) inreg %out) ; ; GFX90a-LABEL: preload_block_count_xyz: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x10 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB11_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB11_0: ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s8 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: v_mov_b32_e32 v2, s10 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v0, s10 +; GFX90a-NEXT: v_mov_b32_e32 v1, s11 +; GFX90a-NEXT: v_mov_b32_e32 v2, s12 +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep_x = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 0 @@ -454,17 +449,17 @@ define amdgpu_kernel void @preload_workgroup_size_x(ptr addrspace(1) inreg %out) ; ; GFX90a-LABEL: preload_workgroup_size_x: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB12_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB12_0: -; GFX90a-NEXT: s_and_b32 s0, s11, 0xffff +; GFX90a-NEXT: s_and_b32 s0, s13, 0xffff ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 12 @@ -492,17 +487,17 @@ define amdgpu_kernel void @preload_workgroup_size_y(ptr addrspace(1) inreg %out) ; ; GFX90a-LABEL: preload_workgroup_size_y: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB13_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB13_0: -; GFX90a-NEXT: s_lshr_b32 s0, s11, 16 +; GFX90a-NEXT: s_lshr_b32 s0, s13, 16 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 14 @@ -531,18 +526,18 @@ define amdgpu_kernel void @preload_workgroup_size_z(ptr addrspace(1) inreg %out) ; ; GFX90a-LABEL: preload_workgroup_size_z: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 +; GFX90a-NEXT: s_load_dword s14, s[4:5], 0x18 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB14_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB14_0: -; GFX90a-NEXT: s_and_b32 s0, s12, 0xffff +; GFX90a-NEXT: s_and_b32 s0, s14, 0xffff ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 16 @@ -575,22 +570,22 @@ define amdgpu_kernel void @preload_workgroup_size_xyz(ptr addrspace(1) inreg %ou ; ; GFX90a-LABEL: preload_workgroup_size_xyz: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 +; GFX90a-NEXT: s_load_dword s14, s[4:5], 0x18 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB15_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB15_0: -; GFX90a-NEXT: s_lshr_b32 s0, s11, 16 -; GFX90a-NEXT: s_and_b32 s1, s11, 0xffff -; GFX90a-NEXT: s_and_b32 s2, s12, 0xffff +; GFX90a-NEXT: s_lshr_b32 s0, s13, 16 +; GFX90a-NEXT: s_and_b32 s1, s13, 0xffff +; GFX90a-NEXT: s_and_b32 s2, s14, 0xffff ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 ; GFX90a-NEXT: v_mov_b32_e32 v0, s1 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 ; GFX90a-NEXT: v_mov_b32_e32 v2, s2 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep_x = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 12 @@ -628,18 +623,18 @@ define amdgpu_kernel void @preload_remainder_x(ptr addrspace(1) inreg %out) #0 { ; ; GFX90a-LABEL: preload_remainder_x: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 +; GFX90a-NEXT: s_load_dword s14, s[4:5], 0x18 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB16_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB16_0: -; GFX90a-NEXT: s_lshr_b32 s0, s12, 16 +; GFX90a-NEXT: s_lshr_b32 s0, s14, 16 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 18 @@ -668,18 +663,16 @@ define amdgpu_kernel void @preloadremainder_y(ptr addrspace(1) inreg %out) #0 { ; ; GFX90a-LABEL: preloadremainder_y: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB17_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB17_0: -; GFX90a-NEXT: s_and_b32 s0, s13, 0xffff +; GFX90a-NEXT: s_and_b32 s0, s15, 0xffff ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 20 @@ -708,18 +701,16 @@ define amdgpu_kernel void @preloadremainder_z(ptr addrspace(1) inreg %out) #0 { ; ; GFX90a-LABEL: preloadremainder_z: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB18_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB18_0: -; GFX90a-NEXT: s_lshr_b32 s0, s13, 16 +; GFX90a-NEXT: s_lshr_b32 s0, s15, 16 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 22 @@ -752,22 +743,20 @@ define amdgpu_kernel void @preloadremainder_xyz(ptr addrspace(1) inreg %out) #0 ; ; GFX90a-LABEL: preloadremainder_xyz: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB19_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB19_0: -; GFX90a-NEXT: s_lshr_b32 s0, s13, 16 -; GFX90a-NEXT: s_lshr_b32 s1, s12, 16 -; GFX90a-NEXT: s_and_b32 s2, s13, 0xffff +; GFX90a-NEXT: s_lshr_b32 s0, s15, 16 +; GFX90a-NEXT: s_lshr_b32 s1, s14, 16 +; GFX90a-NEXT: s_and_b32 s2, s15, 0xffff ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 ; GFX90a-NEXT: v_mov_b32_e32 v0, s1 ; GFX90a-NEXT: v_mov_b32_e32 v1, s2 ; GFX90a-NEXT: v_mov_b32_e32 v2, s0 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep_x = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 18 @@ -803,7 +792,7 @@ define amdgpu_kernel void @no_free_sgprs_preloadremainder_z(ptr addrspace(1) inr ; ; GFX90a-LABEL: no_free_sgprs_preloadremainder_z: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[8:9], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[14:15], s[8:9], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB20_0 ; GFX90a-NEXT: .p2align 8 @@ -814,7 +803,7 @@ define amdgpu_kernel void @no_free_sgprs_preloadremainder_z(ptr addrspace(1) inr ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_lshr_b32 s0, s0, 16 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[12:13] +; GFX90a-NEXT: global_store_dword v0, v1, s[14:15] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 22 @@ -844,10 +833,7 @@ define amdgpu_kernel void @preload_block_max_user_sgprs(ptr addrspace(1) inreg % ; ; GFX90a-LABEL: preload_block_max_user_sgprs: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 -; GFX90a-NEXT: s_load_dword s14, s[4:5], 0x20 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB21_0 ; GFX90a-NEXT: .p2align 8 @@ -857,7 +843,7 @@ define amdgpu_kernel void @preload_block_max_user_sgprs(ptr addrspace(1) inreg % ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %load = load i32, ptr addrspace(4) %imp_arg_ptr @@ -887,21 +873,23 @@ define amdgpu_kernel void @preload_block_count_z_workgroup_size_z_remainder_z(pt ; ; GFX90a-LABEL: preload_block_count_z_workgroup_size_z_remainder_z: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 +; GFX90a-NEXT: s_load_dword s14, s[4:5], 0x18 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB22_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB22_0: -; GFX90a-NEXT: s_lshr_b32 s0, s13, 16 -; GFX90a-NEXT: s_and_b32 s1, s12, 0xffff +; GFX90a-NEXT: s_load_dword s0, s[4:5], 0x1c +; GFX90a-NEXT: s_and_b32 s1, s14, 0xffff ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s10 +; GFX90a-NEXT: v_mov_b32_e32 v0, s12 ; GFX90a-NEXT: v_mov_b32_e32 v1, s1 +; GFX90a-NEXT: s_waitcnt lgkmcnt(0) +; GFX90a-NEXT: s_lshr_b32 s0, s0, 16 ; GFX90a-NEXT: v_mov_b32_e32 v2, s0 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm %imp_arg_ptr = call ptr addrspace(4) @llvm.amdgcn.implicitarg.ptr() %gep0 = getelementptr i8, ptr addrspace(4) %imp_arg_ptr, i32 8 diff --git a/llvm/test/CodeGen/AMDGPU/preload-kernargs.ll b/llvm/test/CodeGen/AMDGPU/preload-kernargs.ll index fe6378435a42e..7ae0c11dca279 100644 --- a/llvm/test/CodeGen/AMDGPU/preload-kernargs.ll +++ b/llvm/test/CodeGen/AMDGPU/preload-kernargs.ll @@ -21,17 +21,17 @@ define amdgpu_kernel void @ptr1_i8(ptr addrspace(1) inreg %out, i8 inreg %arg0) ; ; GFX90a-LABEL: ptr1_i8: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB0_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB0_0: -; GFX90a-NEXT: s_and_b32 s0, s8, 0xff +; GFX90a-NEXT: s_and_b32 s0, s10, 0xff ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %ext = zext i8 %arg0 to i32 store i32 %ext, ptr addrspace(1) %out @@ -56,17 +56,17 @@ define amdgpu_kernel void @ptr1_i8_zext_arg(ptr addrspace(1) inreg %out, i8 zero ; ; GFX90a-LABEL: ptr1_i8_zext_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB1_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB1_0: -; GFX90a-NEXT: s_and_b32 s0, s8, 0xff +; GFX90a-NEXT: s_and_b32 s0, s10, 0xff ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %ext = zext i8 %arg0 to i32 store i32 %ext, ptr addrspace(1) %out, align 4 @@ -91,17 +91,17 @@ define amdgpu_kernel void @ptr1_i16_preload_arg(ptr addrspace(1) inreg %out, i16 ; ; GFX90a-LABEL: ptr1_i16_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB2_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB2_0: -; GFX90a-NEXT: s_and_b32 s0, s8, 0xffff +; GFX90a-NEXT: s_and_b32 s0, s10, 0xffff ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %ext = zext i16 %arg0 to i32 store i32 %ext, ptr addrspace(1) %out, align 4 @@ -125,16 +125,16 @@ define amdgpu_kernel void @ptr1_i32_preload_arg(ptr addrspace(1) inreg %out, i32 ; ; GFX90a-LABEL: ptr1_i32_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB3_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB3_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store i32 %arg0, ptr addrspace(1) %out ret void @@ -160,18 +160,17 @@ define amdgpu_kernel void @i32_ptr1_i32_preload_arg(i32 inreg %arg0, ptr addrspa ; ; GFX90a-LABEL: i32_ptr1_i32_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x10 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s12, s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB4_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB4_0: -; GFX90a-NEXT: s_add_i32 s0, s6, s10 +; GFX90a-NEXT: s_add_i32 s0, s8, s12 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] +; GFX90a-NEXT: global_store_dword v0, v1, s[10:11] ; GFX90a-NEXT: s_endpgm %add = add i32 %arg0, %arg1 store i32 %add, ptr addrspace(1) %out @@ -198,19 +197,19 @@ define amdgpu_kernel void @ptr1_i16_i16_preload_arg(ptr addrspace(1) inreg %out, ; ; GFX90a-LABEL: ptr1_i16_i16_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB5_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB5_0: -; GFX90a-NEXT: s_lshr_b32 s0, s8, 16 -; GFX90a-NEXT: s_and_b32 s1, s8, 0xffff +; GFX90a-NEXT: s_lshr_b32 s0, s10, 16 +; GFX90a-NEXT: s_and_b32 s1, s10, 0xffff ; GFX90a-NEXT: s_add_i32 s0, s1, s0 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %ext = zext i16 %arg0 to i32 %ext1 = zext i16 %arg1 to i32 @@ -236,16 +235,16 @@ define amdgpu_kernel void @ptr1_v2i8_preload_arg(ptr addrspace(1) inreg %out, <2 ; ; GFX90a-LABEL: ptr1_v2i8_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB6_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB6_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store <2 x i8> %in, ptr addrspace(1) %out ret void @@ -274,7 +273,7 @@ define amdgpu_kernel void @byref_preload_arg(ptr addrspace(1) inreg %out, ptr ad ; ; GFX90a-LABEL: byref_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB7_0 ; GFX90a-NEXT: .p2align 8 @@ -285,9 +284,9 @@ define amdgpu_kernel void @byref_preload_arg(ptr addrspace(1) inreg %out, ptr ad ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 ; GFX90a-NEXT: v_mov_b32_e32 v2, s1 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_waitcnt vmcnt(0) -; GFX90a-NEXT: global_store_dword v0, v2, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v2, s[8:9] ; GFX90a-NEXT: s_waitcnt vmcnt(0) ; GFX90a-NEXT: s_endpgm %in = load i32, ptr addrspace(4) %in.byref @@ -320,7 +319,7 @@ define amdgpu_kernel void @byref_staggered_preload_arg(ptr addrspace(1) inreg %o ; ; GFX90a-LABEL: byref_staggered_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB8_0 ; GFX90a-NEXT: .p2align 8 @@ -331,9 +330,9 @@ define amdgpu_kernel void @byref_staggered_preload_arg(ptr addrspace(1) inreg %o ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 ; GFX90a-NEXT: v_mov_b32_e32 v2, s1 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_waitcnt vmcnt(0) -; GFX90a-NEXT: global_store_dword v0, v2, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v2, s[8:9] ; GFX90a-NEXT: s_waitcnt vmcnt(0) ; GFX90a-NEXT: s_endpgm %in = load i32, ptr addrspace(4) %in.byref @@ -370,26 +369,26 @@ define amdgpu_kernel void @v8i32_arg(ptr addrspace(1) nocapture inreg %out, <8 x ; ; GFX90a-LABEL: v8i32_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB9_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB9_0: -; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x20 +; GFX90a-NEXT: s_load_dwordx8 s[12:19], s[4:5], 0x20 ; GFX90a-NEXT: v_mov_b32_e32 v4, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) +; GFX90a-NEXT: v_mov_b32_e32 v0, s16 +; GFX90a-NEXT: v_mov_b32_e32 v1, s17 +; GFX90a-NEXT: v_mov_b32_e32 v2, s18 +; GFX90a-NEXT: v_mov_b32_e32 v3, s19 +; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[8:9] offset:16 +; GFX90a-NEXT: s_nop 0 ; GFX90a-NEXT: v_mov_b32_e32 v0, s12 ; GFX90a-NEXT: v_mov_b32_e32 v1, s13 ; GFX90a-NEXT: v_mov_b32_e32 v2, s14 ; GFX90a-NEXT: v_mov_b32_e32 v3, s15 -; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[6:7] offset:16 -; GFX90a-NEXT: s_nop 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s8 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: v_mov_b32_e32 v2, s10 -; GFX90a-NEXT: v_mov_b32_e32 v3, s11 -; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[6:7] +; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[8:9] ; GFX90a-NEXT: s_endpgm store <8 x i32> %in, ptr addrspace(1) %out, align 4 ret void @@ -414,18 +413,17 @@ define amdgpu_kernel void @v3i16_preload_arg(ptr addrspace(1) nocapture inreg %o ; ; GFX90a-LABEL: v3i16_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB10_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB10_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] offset:4 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s11 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] offset:4 +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store <3 x i16> %in, ptr addrspace(1) %out, align 4 ret void @@ -451,19 +449,17 @@ define amdgpu_kernel void @v3i32_preload_arg(ptr addrspace(1) nocapture inreg %o ; ; GFX90a-LABEL: v3i32_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB11_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB11_0: -; GFX90a-NEXT: v_mov_b32_e32 v0, s10 -; GFX90a-NEXT: v_mov_b32_e32 v1, s11 -; GFX90a-NEXT: v_mov_b32_e32 v2, s12 +; GFX90a-NEXT: v_mov_b32_e32 v0, s12 +; GFX90a-NEXT: v_mov_b32_e32 v1, s13 +; GFX90a-NEXT: v_mov_b32_e32 v2, s14 ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm store <3 x i32> %in, ptr addrspace(1) %out, align 4 ret void @@ -489,19 +485,17 @@ define amdgpu_kernel void @v3f32_preload_arg(ptr addrspace(1) nocapture inreg %o ; ; GFX90a-LABEL: v3f32_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB12_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB12_0: ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s10 -; GFX90a-NEXT: v_mov_b32_e32 v1, s11 -; GFX90a-NEXT: v_mov_b32_e32 v2, s12 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v0, s12 +; GFX90a-NEXT: v_mov_b32_e32 v1, s13 +; GFX90a-NEXT: v_mov_b32_e32 v2, s14 +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm store <3 x float> %in, ptr addrspace(1) %out, align 4 ret void @@ -533,25 +527,24 @@ define amdgpu_kernel void @v5i8_preload_arg(ptr addrspace(1) nocapture inreg %ou ; ; GFX90a-LABEL: v5i8_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB13_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB13_0: -; GFX90a-NEXT: s_lshr_b32 s1, s8, 24 +; GFX90a-NEXT: s_lshr_b32 s1, s10, 24 ; GFX90a-NEXT: s_lshl_b32 s1, s1, 8 -; GFX90a-NEXT: s_bfe_u32 s2, s8, 0x80010 +; GFX90a-NEXT: s_bfe_u32 s2, s10, 0x80010 ; GFX90a-NEXT: s_or_b32 s1, s2, s1 -; GFX90a-NEXT: s_and_b32 s0, s8, 0xffff +; GFX90a-NEXT: s_and_b32 s0, s10, 0xffff ; GFX90a-NEXT: s_lshl_b32 s1, s1, 16 ; GFX90a-NEXT: s_or_b32 s0, s0, s1 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: global_store_byte v0, v1, s[6:7] offset:4 +; GFX90a-NEXT: v_mov_b32_e32 v1, s11 +; GFX90a-NEXT: global_store_byte v0, v1, s[8:9] offset:4 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store <5 x i8> %in, ptr addrspace(1) %out, align 4 ret void @@ -587,29 +580,29 @@ define amdgpu_kernel void @v5f64_arg(ptr addrspace(1) nocapture inreg %out, <5 x ; ; GFX90a-LABEL: v5f64_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB14_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB14_0: ; GFX90a-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x60 -; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x40 +; GFX90a-NEXT: s_load_dwordx8 s[12:19], s[4:5], 0x40 ; GFX90a-NEXT: v_mov_b32_e32 v4, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: v_pk_mov_b32 v[2:3], s[0:1], s[0:1] op_sel:[0,1] +; GFX90a-NEXT: v_mov_b32_e32 v0, s16 +; GFX90a-NEXT: global_store_dwordx2 v4, v[2:3], s[8:9] offset:32 +; GFX90a-NEXT: v_mov_b32_e32 v1, s17 +; GFX90a-NEXT: v_mov_b32_e32 v2, s18 +; GFX90a-NEXT: v_mov_b32_e32 v3, s19 +; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[8:9] offset:16 +; GFX90a-NEXT: s_nop 0 ; GFX90a-NEXT: v_mov_b32_e32 v0, s12 -; GFX90a-NEXT: global_store_dwordx2 v4, v[2:3], s[6:7] offset:32 ; GFX90a-NEXT: v_mov_b32_e32 v1, s13 ; GFX90a-NEXT: v_mov_b32_e32 v2, s14 ; GFX90a-NEXT: v_mov_b32_e32 v3, s15 -; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[6:7] offset:16 -; GFX90a-NEXT: s_nop 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s8 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: v_mov_b32_e32 v2, s10 -; GFX90a-NEXT: v_mov_b32_e32 v3, s11 -; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[6:7] +; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[8:9] ; GFX90a-NEXT: s_endpgm store <5 x double> %in, ptr addrspace(1) %out, align 8 ret void @@ -647,31 +640,30 @@ define amdgpu_kernel void @v8i8_preload_arg(ptr addrspace(1) inreg %out, <8 x i8 ; ; GFX90a-LABEL: v8i8_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB15_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB15_0: -; GFX90a-NEXT: s_lshr_b32 s1, s9, 24 +; GFX90a-NEXT: s_lshr_b32 s1, s11, 24 ; GFX90a-NEXT: s_lshl_b32 s1, s1, 8 -; GFX90a-NEXT: s_bfe_u32 s2, s9, 0x80010 +; GFX90a-NEXT: s_bfe_u32 s2, s11, 0x80010 ; GFX90a-NEXT: s_or_b32 s1, s2, s1 -; GFX90a-NEXT: s_lshr_b32 s2, s8, 24 +; GFX90a-NEXT: s_lshr_b32 s2, s10, 24 ; GFX90a-NEXT: s_lshl_b32 s2, s2, 8 -; GFX90a-NEXT: s_bfe_u32 s3, s8, 0x80010 -; GFX90a-NEXT: s_and_b32 s0, s9, 0xffff +; GFX90a-NEXT: s_bfe_u32 s3, s10, 0x80010 +; GFX90a-NEXT: s_and_b32 s0, s11, 0xffff ; GFX90a-NEXT: s_lshl_b32 s1, s1, 16 ; GFX90a-NEXT: s_or_b32 s2, s3, s2 ; GFX90a-NEXT: s_or_b32 s0, s0, s1 -; GFX90a-NEXT: s_and_b32 s1, s8, 0xffff +; GFX90a-NEXT: s_and_b32 s1, s10, 0xffff ; GFX90a-NEXT: s_lshl_b32 s2, s2, 16 ; GFX90a-NEXT: s_or_b32 s1, s1, s2 ; GFX90a-NEXT: v_mov_b32_e32 v0, s1 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 ; GFX90a-NEXT: v_mov_b32_e32 v2, 0 -; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[6:7] +; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[8:9] ; GFX90a-NEXT: s_endpgm store <8 x i8> %in, ptr addrspace(1) %out ret void @@ -694,16 +686,15 @@ define amdgpu_kernel void @i64_kernel_preload_arg(ptr addrspace(1) inreg %out, i ; ; GFX90a-LABEL: i64_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB16_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB16_0: ; GFX90a-NEXT: v_mov_b32_e32 v2, 0 -; GFX90a-NEXT: v_pk_mov_b32 v[0:1], s[8:9], s[8:9] op_sel:[0,1] -; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[6:7] +; GFX90a-NEXT: v_pk_mov_b32 v[0:1], s[10:11], s[10:11] op_sel:[0,1] +; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[8:9] ; GFX90a-NEXT: s_endpgm store i64 %a, ptr addrspace(1) %out, align 8 ret void @@ -726,16 +717,15 @@ define amdgpu_kernel void @f64_kernel_preload_arg(ptr addrspace(1) inreg %out, d ; ; GFX90a-LABEL: f64_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB17_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB17_0: ; GFX90a-NEXT: v_mov_b32_e32 v2, 0 -; GFX90a-NEXT: v_pk_mov_b32 v[0:1], s[8:9], s[8:9] op_sel:[0,1] -; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[6:7] +; GFX90a-NEXT: v_pk_mov_b32 v[0:1], s[10:11], s[10:11] op_sel:[0,1] +; GFX90a-NEXT: global_store_dwordx2 v2, v[0:1], s[8:9] ; GFX90a-NEXT: s_endpgm store double %in, ptr addrspace(1) %out ret void @@ -758,16 +748,16 @@ define amdgpu_kernel void @half_kernel_preload_arg(ptr addrspace(1) inreg %out, ; ; GFX90a-LABEL: half_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB18_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB18_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store half %in, ptr addrspace(1) %out ret void @@ -790,16 +780,16 @@ define amdgpu_kernel void @bfloat_kernel_preload_arg(ptr addrspace(1) inreg %out ; ; GFX90a-LABEL: bfloat_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB19_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB19_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store bfloat %in, ptr addrspace(1) %out ret void @@ -822,16 +812,16 @@ define amdgpu_kernel void @v2bfloat_kernel_preload_arg(ptr addrspace(1) inreg %o ; ; GFX90a-LABEL: v2bfloat_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB20_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB20_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store <2 x bfloat> %in, ptr addrspace(1) %out ret void @@ -856,18 +846,17 @@ define amdgpu_kernel void @v3bfloat_kernel_preload_arg(ptr addrspace(1) inreg %o ; ; GFX90a-LABEL: v3bfloat_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB21_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB21_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] offset:4 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s11 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] offset:4 +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store <3 x bfloat> %in, ptr addrspace(1) %out ret void @@ -893,19 +882,17 @@ define amdgpu_kernel void @v6bfloat_kernel_preload_arg(ptr addrspace(1) inreg %o ; ; GFX90a-LABEL: v6bfloat_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB22_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB22_0: -; GFX90a-NEXT: v_mov_b32_e32 v0, s10 -; GFX90a-NEXT: v_mov_b32_e32 v1, s11 -; GFX90a-NEXT: v_mov_b32_e32 v2, s12 +; GFX90a-NEXT: v_mov_b32_e32 v0, s12 +; GFX90a-NEXT: v_mov_b32_e32 v1, s13 +; GFX90a-NEXT: v_mov_b32_e32 v2, s14 ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm store <6 x bfloat> %in, ptr addrspace(1) %out ret void @@ -934,24 +921,24 @@ define amdgpu_kernel void @half_v7bfloat_kernel_preload_arg(ptr addrspace(1) inr ; ; GFX90a-LABEL: half_v7bfloat_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB23_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB23_0: -; GFX90a-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x20 +; GFX90a-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x10 +; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x20 ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s8 -; GFX90a-NEXT: global_store_short v3, v0, s[6:7] -; GFX90a-NEXT: v_mov_b32_e32 v0, s13 -; GFX90a-NEXT: s_waitcnt lgkmcnt(0) -; GFX90a-NEXT: global_store_short v3, v0, s[0:1] offset:12 -; GFX90a-NEXT: v_mov_b32_e32 v2, s12 ; GFX90a-NEXT: v_mov_b32_e32 v0, s10 -; GFX90a-NEXT: v_mov_b32_e32 v1, s11 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[0:1] +; GFX90a-NEXT: global_store_short v3, v0, s[8:9] +; GFX90a-NEXT: s_waitcnt lgkmcnt(0) +; GFX90a-NEXT: v_mov_b32_e32 v0, s3 +; GFX90a-NEXT: global_store_short v3, v0, s[6:7] offset:12 +; GFX90a-NEXT: v_mov_b32_e32 v2, s2 +; GFX90a-NEXT: v_mov_b32_e32 v0, s0 +; GFX90a-NEXT: v_mov_b32_e32 v1, s1 +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] ; GFX90a-NEXT: s_endpgm store half %in, ptr addrspace(1) %out store <7 x bfloat> %in2, ptr addrspace(1) %out2 @@ -976,17 +963,17 @@ define amdgpu_kernel void @i1_kernel_preload_arg(ptr addrspace(1) inreg %out, i1 ; ; GFX90a-LABEL: i1_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s10, s[4:5], 0x8 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB24_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB24_0: -; GFX90a-NEXT: s_and_b32 s0, s8, 1 +; GFX90a-NEXT: s_and_b32 s0, s10, 1 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_byte v0, v1, s[6:7] +; GFX90a-NEXT: global_store_byte v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store i1 %in, ptr addrspace(1) %out ret void @@ -1013,20 +1000,18 @@ define amdgpu_kernel void @fp128_kernel_preload_arg(ptr addrspace(1) inreg %out, ; ; GFX90a-LABEL: fp128_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB25_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB25_0: ; GFX90a-NEXT: v_mov_b32_e32 v4, 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s10 -; GFX90a-NEXT: v_mov_b32_e32 v1, s11 -; GFX90a-NEXT: v_mov_b32_e32 v2, s12 -; GFX90a-NEXT: v_mov_b32_e32 v3, s13 -; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v0, s12 +; GFX90a-NEXT: v_mov_b32_e32 v1, s13 +; GFX90a-NEXT: v_mov_b32_e32 v2, s14 +; GFX90a-NEXT: v_mov_b32_e32 v3, s15 +; GFX90a-NEXT: global_store_dwordx4 v4, v[0:3], s[8:9] ; GFX90a-NEXT: s_endpgm store fp128 %in, ptr addrspace(1) %out ret void @@ -1059,26 +1044,25 @@ define amdgpu_kernel void @v7i8_kernel_preload_arg(ptr addrspace(1) inreg %out, ; ; GFX90a-LABEL: v7i8_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB26_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB26_0: -; GFX90a-NEXT: s_lshr_b32 s1, s8, 24 +; GFX90a-NEXT: s_lshr_b32 s1, s10, 24 ; GFX90a-NEXT: s_lshl_b32 s1, s1, 8 -; GFX90a-NEXT: s_bfe_u32 s2, s8, 0x80010 +; GFX90a-NEXT: s_bfe_u32 s2, s10, 0x80010 ; GFX90a-NEXT: s_or_b32 s1, s2, s1 -; GFX90a-NEXT: s_and_b32 s0, s8, 0xffff +; GFX90a-NEXT: s_and_b32 s0, s10, 0xffff ; GFX90a-NEXT: s_lshl_b32 s1, s1, 16 ; GFX90a-NEXT: s_or_b32 s0, s0, s1 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: global_store_byte_d16_hi v0, v1, s[6:7] offset:6 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] offset:4 +; GFX90a-NEXT: v_mov_b32_e32 v1, s11 +; GFX90a-NEXT: global_store_byte_d16_hi v0, v1, s[8:9] offset:6 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] offset:4 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm store <7 x i8> %in, ptr addrspace(1) %out ret void @@ -1106,21 +1090,19 @@ define amdgpu_kernel void @v7half_kernel_preload_arg(ptr addrspace(1) inreg %out ; ; GFX90a-LABEL: v7half_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 -; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x18 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB27_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB27_0: ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: v_mov_b32_e32 v0, s13 -; GFX90a-NEXT: global_store_short v3, v0, s[6:7] offset:12 -; GFX90a-NEXT: v_mov_b32_e32 v2, s12 -; GFX90a-NEXT: v_mov_b32_e32 v0, s10 -; GFX90a-NEXT: v_mov_b32_e32 v1, s11 -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v0, s15 +; GFX90a-NEXT: global_store_short v3, v0, s[8:9] offset:12 +; GFX90a-NEXT: v_mov_b32_e32 v2, s14 +; GFX90a-NEXT: v_mov_b32_e32 v0, s12 +; GFX90a-NEXT: v_mov_b32_e32 v1, s13 +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[8:9] ; GFX90a-NEXT: s_endpgm store <7 x half> %in, ptr addrspace(1) %out ret void @@ -1145,18 +1127,18 @@ define amdgpu_kernel void @i16_i32_kernel_preload_arg(ptr addrspace(1) inreg %ou ; ; GFX90a-LABEL: i16_i32_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB28_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB28_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] -; GFX90a-NEXT: v_mov_b32_e32 v1, s9 -; GFX90a-NEXT: global_store_dword v0, v1, s[10:11] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] +; GFX90a-NEXT: v_mov_b32_e32 v1, s11 +; GFX90a-NEXT: global_store_dword v0, v1, s[12:13] ; GFX90a-NEXT: s_endpgm store i16 %in, ptr addrspace(1) %out store i32 %in2, ptr addrspace(1) %out2 @@ -1184,22 +1166,22 @@ define amdgpu_kernel void @i16_v3i32_kernel_preload_arg(ptr addrspace(1) inreg % ; ; GFX90a-LABEL: i16_v3i32_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx8 s[8:15], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB29_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB29_0: -; GFX90a-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x20 +; GFX90a-NEXT: s_load_dwordx4 s[0:3], s[4:5], 0x10 ; GFX90a-NEXT: v_mov_b32_e32 v3, 0 -; GFX90a-NEXT: v_mov_b32_e32 v4, s8 -; GFX90a-NEXT: v_mov_b32_e32 v0, s10 -; GFX90a-NEXT: v_mov_b32_e32 v1, s11 -; GFX90a-NEXT: v_mov_b32_e32 v2, s12 -; GFX90a-NEXT: global_store_short v3, v4, s[6:7] +; GFX90a-NEXT: s_load_dwordx2 s[4:5], s[4:5], 0x20 +; GFX90a-NEXT: v_mov_b32_e32 v4, s10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) -; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[0:1] +; GFX90a-NEXT: v_mov_b32_e32 v0, s0 +; GFX90a-NEXT: v_mov_b32_e32 v1, s1 +; GFX90a-NEXT: v_mov_b32_e32 v2, s2 +; GFX90a-NEXT: global_store_short v3, v4, s[8:9] +; GFX90a-NEXT: global_store_dwordx3 v3, v[0:2], s[4:5] ; GFX90a-NEXT: s_endpgm store i16 %in, ptr addrspace(1) %out store <3 x i32> %in2, ptr addrspace(1) %out2 @@ -1224,17 +1206,17 @@ define amdgpu_kernel void @i16_i16_kernel_preload_arg(ptr addrspace(1) inreg %ou ; ; GFX90a-LABEL: i16_i16_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB30_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB30_0: ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] -; GFX90a-NEXT: global_store_short_d16_hi v0, v1, s[10:11] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] +; GFX90a-NEXT: global_store_short_d16_hi v0, v1, s[12:13] ; GFX90a-NEXT: s_endpgm store i16 %in, ptr addrspace(1) %out store i16 %in2, ptr addrspace(1) %out2 @@ -1264,22 +1246,22 @@ define amdgpu_kernel void @i16_v2i8_kernel_preload_arg(ptr addrspace(1) inreg %o ; ; GFX90a-LABEL: i16_v2i8_kernel_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 +; GFX90a-NEXT: s_load_dwordx2 s[12:13], s[4:5], 0x10 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB31_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB31_0: -; GFX90a-NEXT: s_lshr_b32 s0, s8, 24 +; GFX90a-NEXT: s_lshr_b32 s0, s10, 24 ; GFX90a-NEXT: s_lshl_b32 s0, s0, 8 -; GFX90a-NEXT: s_bfe_u32 s1, s8, 0x80010 +; GFX90a-NEXT: s_bfe_u32 s1, s10, 0x80010 ; GFX90a-NEXT: s_or_b32 s0, s1, s0 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 -; GFX90a-NEXT: v_mov_b32_e32 v1, s8 -; GFX90a-NEXT: global_store_short v0, v1, s[6:7] +; GFX90a-NEXT: v_mov_b32_e32 v1, s10 +; GFX90a-NEXT: global_store_short v0, v1, s[8:9] ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_short v0, v1, s[10:11] +; GFX90a-NEXT: global_store_short v0, v1, s[12:13] ; GFX90a-NEXT: s_endpgm store i16 %in, ptr addrspace(1) %out store <2 x i8> %in2, ptr addrspace(1) %out2 @@ -1308,7 +1290,7 @@ define amdgpu_kernel void @i32_ptr1_i32_staggered_preload_arg(i32 inreg %arg0, p ; ; GFX90a-LABEL: i32_ptr1_i32_staggered_preload_arg: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dword s6, s[4:5], 0x0 +; GFX90a-NEXT: s_load_dword s8, s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB32_0 ; GFX90a-NEXT: .p2align 8 @@ -1318,7 +1300,7 @@ define amdgpu_kernel void @i32_ptr1_i32_staggered_preload_arg(i32 inreg %arg0, p ; GFX90a-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x8 ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) -; GFX90a-NEXT: s_add_i32 s2, s6, s2 +; GFX90a-NEXT: s_add_i32 s2, s8, s2 ; GFX90a-NEXT: v_mov_b32_e32 v1, s2 ; GFX90a-NEXT: global_store_dword v0, v1, s[0:1] ; GFX90a-NEXT: s_endpgm @@ -1345,17 +1327,16 @@ define amdgpu_kernel void @ptr1_i8_trailing_unused(ptr addrspace(1) inreg %out, ; ; GFX90a-LABEL: ptr1_i8_trailing_unused: ; GFX90a: ; %bb.1: -; GFX90a-NEXT: s_load_dwordx2 s[6:7], s[4:5], 0x0 -; GFX90a-NEXT: s_load_dwordx2 s[8:9], s[4:5], 0x8 +; GFX90a-NEXT: s_load_dwordx4 s[8:11], s[4:5], 0x0 ; GFX90a-NEXT: s_waitcnt lgkmcnt(0) ; GFX90a-NEXT: s_branch .LBB33_0 ; GFX90a-NEXT: .p2align 8 ; GFX90a-NEXT: ; %bb.2: ; GFX90a-NEXT: .LBB33_0: -; GFX90a-NEXT: s_and_b32 s0, s8, 0xff +; GFX90a-NEXT: s_and_b32 s0, s10, 0xff ; GFX90a-NEXT: v_mov_b32_e32 v0, 0 ; GFX90a-NEXT: v_mov_b32_e32 v1, s0 -; GFX90a-NEXT: global_store_dword v0, v1, s[6:7] +; GFX90a-NEXT: global_store_dword v0, v1, s[8:9] ; GFX90a-NEXT: s_endpgm %ext = zext i8 %arg0 to i32 store i32 %ext, ptr addrspace(1) %out diff --git a/llvm/test/CodeGen/AMDGPU/sad.ll b/llvm/test/CodeGen/AMDGPU/sad.ll index 5474338514522..8f25e6519588b 100644 --- a/llvm/test/CodeGen/AMDGPU/sad.ll +++ b/llvm/test/CodeGen/AMDGPU/sad.ll @@ -6,6 +6,9 @@ define amdgpu_kernel void @v_sad_u32_pat1(ptr addrspace(1) %out, i32 %a, i32 %b, ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s1 ; GCN-NEXT: v_mov_b32_e32 v1, s2 @@ -33,9 +36,12 @@ define amdgpu_kernel void @v_sad_u32_constant_pat1(ptr addrspace(1) %out, i32 %a ; GCN-NEXT: s_load_dword s2, s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 ; GCN-NEXT: v_mov_b32_e32 v0, 0x5a +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_sad_u32 v2, s2, v0, 20 ; GCN-NEXT: v_mov_b32_e32 v0, s0 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: v_mov_b32_e32 v1, s1 ; GCN-NEXT: flat_store_dword v[0:1], v2 ; GCN-NEXT: s_endpgm @@ -57,6 +63,9 @@ define amdgpu_kernel void @v_sad_u32_pat2(ptr addrspace(1) %out, i32 %a, i32 %b, ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s1 ; GCN-NEXT: v_mov_b32_e32 v1, s2 @@ -79,12 +88,14 @@ define amdgpu_kernel void @v_sad_u32_pat2(ptr addrspace(1) %out, i32 %a, i32 %b, define amdgpu_kernel void @v_sad_u32_multi_use_sub_pat1(ptr addrspace(1) %out, i32 %a, i32 %b, i32 %c) { ; GCN-LABEL: v_sad_u32_multi_use_sub_pat1: ; GCN: ; %bb.0: -; GCN-NEXT: s_mov_b64 s[18:19], s[2:3] -; GCN-NEXT: s_mov_b64 s[16:17], s[0:1] +; GCN-NEXT: s_mov_b64 s[22:23], s[2:3] +; GCN-NEXT: s_mov_b64 s[20:21], s[0:1] ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 -; GCN-NEXT: s_add_u32 s16, s16, s15 -; GCN-NEXT: s_addc_u32 s17, s17, 0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_add_u32 s20, s20, s17 +; GCN-NEXT: s_addc_u32 s21, s21, 0 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_min_u32 s3, s0, s1 ; GCN-NEXT: s_max_u32 s0, s0, s1 @@ -92,8 +103,9 @@ define amdgpu_kernel void @v_sad_u32_multi_use_sub_pat1(ptr addrspace(1) %out, i ; GCN-NEXT: v_mov_b32_e32 v0, s4 ; GCN-NEXT: v_mov_b32_e32 v2, s0 ; GCN-NEXT: s_add_i32 s0, s0, s2 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: v_mov_b32_e32 v1, s5 -; GCN-NEXT: buffer_store_dword v2, v0, s[16:19], 0 offen +; GCN-NEXT: buffer_store_dword v2, v0, s[20:23], 0 offen ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v2, s0 ; GCN-NEXT: flat_store_dword v[0:1], v2 @@ -115,19 +127,22 @@ define amdgpu_kernel void @v_sad_u32_multi_use_sub_pat1(ptr addrspace(1) %out, i define amdgpu_kernel void @v_sad_u32_multi_use_add_pat1(ptr addrspace(1) %out, i32 %a, i32 %b, i32 %c) { ; GCN-LABEL: v_sad_u32_multi_use_add_pat1: ; GCN: ; %bb.0: -; GCN-NEXT: s_mov_b64 s[18:19], s[2:3] -; GCN-NEXT: s_mov_b64 s[16:17], s[0:1] +; GCN-NEXT: s_mov_b64 s[22:23], s[2:3] +; GCN-NEXT: s_mov_b64 s[20:21], s[0:1] ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 -; GCN-NEXT: s_add_u32 s16, s16, s15 -; GCN-NEXT: s_addc_u32 s17, s17, 0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_add_u32 s20, s20, s17 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v2, s1 ; GCN-NEXT: v_mov_b32_e32 v3, s2 ; GCN-NEXT: v_mov_b32_e32 v0, s4 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_addc_u32 s21, s21, 0 ; GCN-NEXT: v_mov_b32_e32 v1, s5 ; GCN-NEXT: v_sad_u32 v2, s0, v2, v3 -; GCN-NEXT: buffer_store_dword v2, v0, s[16:19], 0 offen +; GCN-NEXT: buffer_store_dword v2, v0, s[20:23], 0 offen ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: flat_store_dword v[0:1], v2 ; GCN-NEXT: s_endpgm @@ -147,21 +162,24 @@ define amdgpu_kernel void @v_sad_u32_multi_use_add_pat1(ptr addrspace(1) %out, i define amdgpu_kernel void @v_sad_u32_multi_use_max_pat1(ptr addrspace(1) %out, i32 %a, i32 %b, i32 %c) { ; GCN-LABEL: v_sad_u32_multi_use_max_pat1: ; GCN: ; %bb.0: -; GCN-NEXT: s_mov_b64 s[18:19], s[2:3] -; GCN-NEXT: s_mov_b64 s[16:17], s[0:1] +; GCN-NEXT: s_mov_b64 s[22:23], s[2:3] +; GCN-NEXT: s_mov_b64 s[20:21], s[0:1] ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 -; GCN-NEXT: s_add_u32 s16, s16, s15 -; GCN-NEXT: s_addc_u32 s17, s17, 0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_add_u32 s20, s20, s17 +; GCN-NEXT: s_addc_u32 s21, s21, 0 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_max_u32 s3, s0, s1 ; GCN-NEXT: v_mov_b32_e32 v0, s1 ; GCN-NEXT: v_mov_b32_e32 v1, s2 ; GCN-NEXT: v_mov_b32_e32 v2, s3 ; GCN-NEXT: v_sad_u32 v3, s0, v0, v1 -; GCN-NEXT: buffer_store_dword v2, v0, s[16:19], 0 offen +; GCN-NEXT: buffer_store_dword v2, v0, s[20:23], 0 offen ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s4 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: v_mov_b32_e32 v1, s5 ; GCN-NEXT: flat_store_dword v[0:1], v3 ; GCN-NEXT: s_endpgm @@ -182,21 +200,24 @@ define amdgpu_kernel void @v_sad_u32_multi_use_max_pat1(ptr addrspace(1) %out, i define amdgpu_kernel void @v_sad_u32_multi_use_min_pat1(ptr addrspace(1) %out, i32 %a, i32 %b, i32 %c) { ; GCN-LABEL: v_sad_u32_multi_use_min_pat1: ; GCN: ; %bb.0: -; GCN-NEXT: s_mov_b64 s[18:19], s[2:3] -; GCN-NEXT: s_mov_b64 s[16:17], s[0:1] +; GCN-NEXT: s_mov_b64 s[22:23], s[2:3] +; GCN-NEXT: s_mov_b64 s[20:21], s[0:1] ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 -; GCN-NEXT: s_add_u32 s16, s16, s15 -; GCN-NEXT: s_addc_u32 s17, s17, 0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_add_u32 s20, s20, s17 +; GCN-NEXT: s_addc_u32 s21, s21, 0 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_min_u32 s3, s0, s1 ; GCN-NEXT: v_mov_b32_e32 v0, s1 ; GCN-NEXT: v_mov_b32_e32 v1, s2 ; GCN-NEXT: v_mov_b32_e32 v2, s3 ; GCN-NEXT: v_sad_u32 v3, s0, v0, v1 -; GCN-NEXT: buffer_store_dword v2, v0, s[16:19], 0 offen +; GCN-NEXT: buffer_store_dword v2, v0, s[20:23], 0 offen ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s4 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: v_mov_b32_e32 v1, s5 ; GCN-NEXT: flat_store_dword v[0:1], v3 ; GCN-NEXT: s_endpgm @@ -218,21 +239,24 @@ define amdgpu_kernel void @v_sad_u32_multi_use_min_pat1(ptr addrspace(1) %out, i define amdgpu_kernel void @v_sad_u32_multi_use_sub_pat2(ptr addrspace(1) %out, i32 %a, i32 %b, i32 %c) { ; GCN-LABEL: v_sad_u32_multi_use_sub_pat2: ; GCN: ; %bb.0: -; GCN-NEXT: s_mov_b64 s[18:19], s[2:3] -; GCN-NEXT: s_mov_b64 s[16:17], s[0:1] +; GCN-NEXT: s_mov_b64 s[22:23], s[2:3] +; GCN-NEXT: s_mov_b64 s[20:21], s[0:1] ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 -; GCN-NEXT: s_add_u32 s16, s16, s15 -; GCN-NEXT: s_addc_u32 s17, s17, 0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_add_u32 s20, s20, s17 +; GCN-NEXT: s_addc_u32 s21, s21, 0 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_sub_i32 s3, s0, s1 ; GCN-NEXT: v_mov_b32_e32 v0, s1 ; GCN-NEXT: v_mov_b32_e32 v1, s2 ; GCN-NEXT: v_mov_b32_e32 v2, s3 ; GCN-NEXT: v_sad_u32 v3, s0, v0, v1 -; GCN-NEXT: buffer_store_dword v2, v0, s[16:19], 0 offen +; GCN-NEXT: buffer_store_dword v2, v0, s[20:23], 0 offen ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s4 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: v_mov_b32_e32 v1, s5 ; GCN-NEXT: flat_store_dword v[0:1], v3 ; GCN-NEXT: s_endpgm @@ -251,12 +275,14 @@ define amdgpu_kernel void @v_sad_u32_multi_use_sub_pat2(ptr addrspace(1) %out, i define amdgpu_kernel void @v_sad_u32_multi_use_select_pat2(ptr addrspace(1) %out, i32 %a, i32 %b, i32 %c) { ; GCN-LABEL: v_sad_u32_multi_use_select_pat2: ; GCN: ; %bb.0: -; GCN-NEXT: s_mov_b64 s[18:19], s[2:3] -; GCN-NEXT: s_mov_b64 s[16:17], s[0:1] +; GCN-NEXT: s_mov_b64 s[22:23], s[2:3] +; GCN-NEXT: s_mov_b64 s[20:21], s[0:1] ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 -; GCN-NEXT: s_add_u32 s16, s16, s15 -; GCN-NEXT: s_addc_u32 s17, s17, 0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_add_u32 s20, s20, s17 +; GCN-NEXT: s_addc_u32 s21, s21, 0 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_min_u32 s3, s0, s1 ; GCN-NEXT: s_max_u32 s0, s0, s1 @@ -264,8 +290,9 @@ define amdgpu_kernel void @v_sad_u32_multi_use_select_pat2(ptr addrspace(1) %out ; GCN-NEXT: v_mov_b32_e32 v0, s4 ; GCN-NEXT: v_mov_b32_e32 v2, s0 ; GCN-NEXT: s_add_i32 s0, s0, s2 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: v_mov_b32_e32 v1, s5 -; GCN-NEXT: buffer_store_dword v2, v0, s[16:19], 0 offen +; GCN-NEXT: buffer_store_dword v2, v0, s[20:23], 0 offen ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v2, s0 ; GCN-NEXT: flat_store_dword v[0:1], v2 @@ -285,6 +312,9 @@ define amdgpu_kernel void @v_sad_u32_multi_use_select_pat2(ptr addrspace(1) %out define amdgpu_kernel void @v_sad_u32_vector_pat1(ptr addrspace(1) %out, <4 x i32> %a, <4 x i32> %b, <4 x i32> %c) { ; GCN-LABEL: v_sad_u32_vector_pat1: ; GCN: ; %bb.0: +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x4 ; GCN-NEXT: s_load_dwordx4 s[12:15], s[8:9], 0xc ; GCN-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 @@ -321,6 +351,9 @@ define amdgpu_kernel void @v_sad_u32_vector_pat1(ptr addrspace(1) %out, <4 x i32 define amdgpu_kernel void @v_sad_u32_vector_pat2(ptr addrspace(1) %out, <4 x i32> %a, <4 x i32> %b, <4 x i32> %c) { ; GCN-LABEL: v_sad_u32_vector_pat2: ; GCN: ; %bb.0: +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x4 ; GCN-NEXT: s_load_dwordx4 s[12:15], s[8:9], 0xc ; GCN-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 @@ -358,6 +391,8 @@ define amdgpu_kernel void @v_sad_u32_i16_pat1(ptr addrspace(1) %out, i16 %a, i16 ; GCN-NEXT: s_load_dword s4, s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_and_b32 s4, s4, 0xffff ; GCN-NEXT: s_lshr_b32 s0, s0, 16 @@ -365,6 +400,7 @@ define amdgpu_kernel void @v_sad_u32_i16_pat1(ptr addrspace(1) %out, i16 %a, i16 ; GCN-NEXT: v_mov_b32_e32 v1, s0 ; GCN-NEXT: v_sad_u32 v2, s4, v1, v0 ; GCN-NEXT: v_mov_b32_e32 v0, s2 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: v_mov_b32_e32 v1, s3 ; GCN-NEXT: flat_store_short v[0:1], v2 ; GCN-NEXT: s_endpgm @@ -384,6 +420,9 @@ define amdgpu_kernel void @v_sad_u32_i16_pat1(ptr addrspace(1) %out, i16 %a, i16 define amdgpu_kernel void @v_sad_u32_i16_pat2(ptr addrspace(1) %out) { ; GCN-LABEL: v_sad_u32_i16_pat2: ; GCN: ; %bb.0: +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: flat_load_ushort v0, v[0:1] glc ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -416,6 +455,9 @@ define amdgpu_kernel void @v_sad_u32_i8_pat1(ptr addrspace(1) %out, i8 %a, i8 %b ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dword s2, s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_and_b32 s3, s2, 0xff ; GCN-NEXT: s_bfe_u32 s4, s2, 0x80008 @@ -443,6 +485,9 @@ define amdgpu_kernel void @v_sad_u32_i8_pat1(ptr addrspace(1) %out, i8 %a, i8 %b define amdgpu_kernel void @v_sad_u32_i8_pat2(ptr addrspace(1) %out) { ; GCN-LABEL: v_sad_u32_i8_pat2: ; GCN: ; %bb.0: +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: flat_load_ubyte v0, v[0:1] glc ; GCN-NEXT: s_waitcnt vmcnt(0) ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 @@ -475,6 +520,9 @@ define amdgpu_kernel void @s_sad_u32_i8_pat2(ptr addrspace(1) %out, i8 zeroext % ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dword s2, s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_and_b32 s3, s2, 0xff ; GCN-NEXT: s_bfe_u32 s4, s2, 0x80008 @@ -502,6 +550,9 @@ define amdgpu_kernel void @v_sad_u32_mismatched_operands_pat1(ptr addrspace(1) % ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_max_u32 s6, s0, s1 ; GCN-NEXT: s_cmp_le_u32 s0, s1 @@ -531,6 +582,9 @@ define amdgpu_kernel void @v_sad_u32_mismatched_operands_pat2(ptr addrspace(1) % ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x2 ; GCN-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_sub_i32 s3, s0, s3 ; GCN-NEXT: s_sub_i32 s6, s1, s0 diff --git a/llvm/test/CodeGen/AMDGPU/scalar_to_vector.v8i16.ll b/llvm/test/CodeGen/AMDGPU/scalar_to_vector.v8i16.ll index 884ba3fc34dff..29448ab2d822e 100644 --- a/llvm/test/CodeGen/AMDGPU/scalar_to_vector.v8i16.ll +++ b/llvm/test/CodeGen/AMDGPU/scalar_to_vector.v8i16.ll @@ -9,6 +9,8 @@ define amdgpu_kernel void @scalar_to_vector_v8i16(<2 x i32> %in, ptr %out) #0 { ; GFX900: ; %bb.0: ; %entry ; GFX900-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX900-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; GFX900-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX900-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX900-NEXT: s_waitcnt lgkmcnt(0) ; GFX900-NEXT: v_mov_b32_e32 v5, s3 ; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 @@ -24,6 +26,8 @@ define amdgpu_kernel void @scalar_to_vector_v8i16(<2 x i32> %in, ptr %out) #0 { ; GFX906: ; %bb.0: ; %entry ; GFX906-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX906-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; GFX906-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX906-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX906-NEXT: s_waitcnt lgkmcnt(0) ; GFX906-NEXT: v_mov_b32_e32 v5, s3 ; GFX906-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 @@ -39,6 +43,8 @@ define amdgpu_kernel void @scalar_to_vector_v8i16(<2 x i32> %in, ptr %out) #0 { ; GFX908: ; %bb.0: ; %entry ; GFX908-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX908-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; GFX908-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX908-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX908-NEXT: s_waitcnt lgkmcnt(0) ; GFX908-NEXT: v_mov_b32_e32 v5, s3 ; GFX908-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 @@ -55,6 +61,8 @@ define amdgpu_kernel void @scalar_to_vector_v8i16(<2 x i32> %in, ptr %out) #0 { ; GFX90A-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX90A-NEXT: v_and_b32_e32 v4, 0x3ff, v0 ; GFX90A-NEXT: v_lshlrev_b32_e32 v4, 4, v4 +; GFX90A-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NEXT: s_waitcnt lgkmcnt(0) ; GFX90A-NEXT: v_mov_b32_e32 v5, s3 ; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 @@ -88,6 +96,8 @@ define amdgpu_kernel void @scalar_to_vector_v8f16(<2 x float> %in, ptr %out) #0 ; GFX900: ; %bb.0: ; %entry ; GFX900-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX900-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; GFX900-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX900-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX900-NEXT: s_waitcnt lgkmcnt(0) ; GFX900-NEXT: v_mov_b32_e32 v5, s3 ; GFX900-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 @@ -103,6 +113,8 @@ define amdgpu_kernel void @scalar_to_vector_v8f16(<2 x float> %in, ptr %out) #0 ; GFX906: ; %bb.0: ; %entry ; GFX906-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX906-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; GFX906-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX906-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX906-NEXT: s_waitcnt lgkmcnt(0) ; GFX906-NEXT: v_mov_b32_e32 v5, s3 ; GFX906-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 @@ -118,6 +130,8 @@ define amdgpu_kernel void @scalar_to_vector_v8f16(<2 x float> %in, ptr %out) #0 ; GFX908: ; %bb.0: ; %entry ; GFX908-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX908-NEXT: v_lshlrev_b32_e32 v4, 4, v0 +; GFX908-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX908-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX908-NEXT: s_waitcnt lgkmcnt(0) ; GFX908-NEXT: v_mov_b32_e32 v5, s3 ; GFX908-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 @@ -134,6 +148,8 @@ define amdgpu_kernel void @scalar_to_vector_v8f16(<2 x float> %in, ptr %out) #0 ; GFX90A-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; GFX90A-NEXT: v_and_b32_e32 v4, 0x3ff, v0 ; GFX90A-NEXT: v_lshlrev_b32_e32 v4, 4, v4 +; GFX90A-NEXT: s_add_u32 flat_scratch_lo, s12, s17 +; GFX90A-NEXT: s_addc_u32 flat_scratch_hi, s13, 0 ; GFX90A-NEXT: s_waitcnt lgkmcnt(0) ; GFX90A-NEXT: v_mov_b32_e32 v5, s3 ; GFX90A-NEXT: v_add_co_u32_e32 v4, vcc, s2, v4 diff --git a/llvm/test/CodeGen/AMDGPU/scc-clobbered-sgpr-to-vmem-spill.ll b/llvm/test/CodeGen/AMDGPU/scc-clobbered-sgpr-to-vmem-spill.ll index 0ad10437299f4..90dfd5a21d107 100644 --- a/llvm/test/CodeGen/AMDGPU/scc-clobbered-sgpr-to-vmem-spill.ll +++ b/llvm/test/CodeGen/AMDGPU/scc-clobbered-sgpr-to-vmem-spill.ll @@ -20,179 +20,183 @@ define amdgpu_kernel void @kernel0(ptr addrspace(1) %out, i32 %in) #1 { ; CHECK-NEXT: ; def s[2:3] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ; implicit-def: $vgpr22 : SGPR spill to VGPR lane -; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[4:7] -; CHECK-NEXT: ;;#ASMEND +; CHECK-NEXT: s_load_dword s0, s[8:9], 0x8 ; CHECK-NEXT: v_writelane_b32 v22, s2, 0 ; CHECK-NEXT: v_writelane_b32 v22, s3, 1 +; CHECK-NEXT: ;;#ASMSTART +; CHECK-NEXT: ; def s[48:51] +; CHECK-NEXT: ;;#ASMEND +; CHECK-NEXT: ;;#ASMSTART +; CHECK-NEXT: ; def s[4:11] +; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: v_writelane_b32 v22, s4, 2 ; CHECK-NEXT: v_writelane_b32 v22, s5, 3 ; CHECK-NEXT: v_writelane_b32 v22, s6, 4 -; CHECK-NEXT: s_load_dword s0, s[8:9], 0x8 ; CHECK-NEXT: v_writelane_b32 v22, s7, 5 -; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[4:11] -; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v22, s4, 6 -; CHECK-NEXT: v_writelane_b32 v22, s5, 7 -; CHECK-NEXT: v_writelane_b32 v22, s6, 8 -; CHECK-NEXT: v_writelane_b32 v22, s7, 9 -; CHECK-NEXT: v_writelane_b32 v22, s8, 10 -; CHECK-NEXT: v_writelane_b32 v22, s9, 11 -; CHECK-NEXT: v_writelane_b32 v22, s10, 12 -; CHECK-NEXT: v_writelane_b32 v22, s11, 13 +; CHECK-NEXT: v_writelane_b32 v22, s8, 6 +; CHECK-NEXT: v_writelane_b32 v22, s9, 7 +; CHECK-NEXT: v_writelane_b32 v22, s10, 8 +; CHECK-NEXT: v_writelane_b32 v22, s11, 9 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[4:19] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v22, s4, 14 -; CHECK-NEXT: v_writelane_b32 v22, s5, 15 -; CHECK-NEXT: v_writelane_b32 v22, s6, 16 -; CHECK-NEXT: v_writelane_b32 v22, s7, 17 -; CHECK-NEXT: v_writelane_b32 v22, s8, 18 -; CHECK-NEXT: v_writelane_b32 v22, s9, 19 -; CHECK-NEXT: v_writelane_b32 v22, s10, 20 -; CHECK-NEXT: v_writelane_b32 v22, s11, 21 -; CHECK-NEXT: v_writelane_b32 v22, s12, 22 -; CHECK-NEXT: v_writelane_b32 v22, s13, 23 -; CHECK-NEXT: v_writelane_b32 v22, s14, 24 -; CHECK-NEXT: v_writelane_b32 v22, s15, 25 -; CHECK-NEXT: v_writelane_b32 v22, s16, 26 -; CHECK-NEXT: v_writelane_b32 v22, s17, 27 -; CHECK-NEXT: v_writelane_b32 v22, s18, 28 -; CHECK-NEXT: v_writelane_b32 v22, s19, 29 +; CHECK-NEXT: v_writelane_b32 v22, s4, 10 +; CHECK-NEXT: v_writelane_b32 v22, s5, 11 +; CHECK-NEXT: v_writelane_b32 v22, s6, 12 +; CHECK-NEXT: v_writelane_b32 v22, s7, 13 +; CHECK-NEXT: v_writelane_b32 v22, s8, 14 +; CHECK-NEXT: v_writelane_b32 v22, s9, 15 +; CHECK-NEXT: v_writelane_b32 v22, s10, 16 +; CHECK-NEXT: v_writelane_b32 v22, s11, 17 +; CHECK-NEXT: v_writelane_b32 v22, s12, 18 +; CHECK-NEXT: v_writelane_b32 v22, s13, 19 +; CHECK-NEXT: v_writelane_b32 v22, s14, 20 +; CHECK-NEXT: v_writelane_b32 v22, s15, 21 +; CHECK-NEXT: v_writelane_b32 v22, s16, 22 +; CHECK-NEXT: v_writelane_b32 v22, s17, 23 +; CHECK-NEXT: v_writelane_b32 v22, s18, 24 +; CHECK-NEXT: v_writelane_b32 v22, s19, 25 ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[42:43] +; CHECK-NEXT: ; def s[38:39] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[52:55] +; CHECK-NEXT: ; def s[44:47] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[4:11] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v22, s4, 30 -; CHECK-NEXT: v_writelane_b32 v22, s5, 31 -; CHECK-NEXT: v_writelane_b32 v22, s6, 32 -; CHECK-NEXT: v_writelane_b32 v22, s7, 33 -; CHECK-NEXT: v_writelane_b32 v22, s8, 34 -; CHECK-NEXT: v_writelane_b32 v22, s9, 35 -; CHECK-NEXT: v_writelane_b32 v22, s10, 36 -; CHECK-NEXT: v_writelane_b32 v22, s11, 37 +; CHECK-NEXT: v_writelane_b32 v22, s4, 26 +; CHECK-NEXT: v_writelane_b32 v22, s5, 27 +; CHECK-NEXT: v_writelane_b32 v22, s6, 28 +; CHECK-NEXT: v_writelane_b32 v22, s7, 29 +; CHECK-NEXT: v_writelane_b32 v22, s8, 30 +; CHECK-NEXT: v_writelane_b32 v22, s9, 31 +; CHECK-NEXT: v_writelane_b32 v22, s10, 32 +; CHECK-NEXT: v_writelane_b32 v22, s11, 33 ; CHECK-NEXT: s_waitcnt lgkmcnt(0) ; CHECK-NEXT: s_cmp_lg_u32 s0, 0 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[16:31] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[40:41] +; CHECK-NEXT: ; def s[36:37] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[36:39] +; CHECK-NEXT: ; def s[40:43] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[44:51] +; CHECK-NEXT: ; def s[0:7] ; CHECK-NEXT: ;;#ASMEND +; CHECK-NEXT: v_writelane_b32 v22, s0, 34 +; CHECK-NEXT: v_writelane_b32 v22, s1, 35 +; CHECK-NEXT: v_writelane_b32 v22, s2, 36 +; CHECK-NEXT: v_writelane_b32 v22, s3, 37 +; CHECK-NEXT: v_writelane_b32 v22, s4, 38 +; CHECK-NEXT: v_writelane_b32 v22, s5, 39 +; CHECK-NEXT: v_writelane_b32 v22, s6, 40 +; CHECK-NEXT: v_writelane_b32 v22, s7, 41 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[0:15] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v22, s0, 38 -; CHECK-NEXT: v_writelane_b32 v22, s1, 39 -; CHECK-NEXT: v_writelane_b32 v22, s2, 40 -; CHECK-NEXT: v_writelane_b32 v22, s3, 41 -; CHECK-NEXT: v_writelane_b32 v22, s4, 42 -; CHECK-NEXT: v_writelane_b32 v22, s5, 43 -; CHECK-NEXT: v_writelane_b32 v22, s6, 44 -; CHECK-NEXT: v_writelane_b32 v22, s7, 45 -; CHECK-NEXT: v_writelane_b32 v22, s8, 46 -; CHECK-NEXT: v_writelane_b32 v22, s9, 47 -; CHECK-NEXT: v_writelane_b32 v22, s10, 48 -; CHECK-NEXT: v_writelane_b32 v22, s11, 49 -; CHECK-NEXT: v_writelane_b32 v22, s12, 50 -; CHECK-NEXT: v_writelane_b32 v22, s13, 51 -; CHECK-NEXT: v_writelane_b32 v22, s14, 52 -; CHECK-NEXT: v_writelane_b32 v22, s15, 53 +; CHECK-NEXT: v_writelane_b32 v22, s0, 42 +; CHECK-NEXT: v_writelane_b32 v22, s1, 43 +; CHECK-NEXT: v_writelane_b32 v22, s2, 44 +; CHECK-NEXT: v_writelane_b32 v22, s3, 45 +; CHECK-NEXT: v_writelane_b32 v22, s4, 46 +; CHECK-NEXT: v_writelane_b32 v22, s5, 47 +; CHECK-NEXT: v_writelane_b32 v22, s6, 48 +; CHECK-NEXT: v_writelane_b32 v22, s7, 49 +; CHECK-NEXT: v_writelane_b32 v22, s8, 50 +; CHECK-NEXT: v_writelane_b32 v22, s9, 51 +; CHECK-NEXT: v_writelane_b32 v22, s10, 52 +; CHECK-NEXT: v_writelane_b32 v22, s11, 53 +; CHECK-NEXT: v_writelane_b32 v22, s12, 54 +; CHECK-NEXT: v_writelane_b32 v22, s13, 55 +; CHECK-NEXT: v_writelane_b32 v22, s14, 56 +; CHECK-NEXT: v_writelane_b32 v22, s15, 57 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[34:35] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[0:3] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v22, s0, 54 -; CHECK-NEXT: v_writelane_b32 v22, s1, 55 -; CHECK-NEXT: v_writelane_b32 v22, s2, 56 -; CHECK-NEXT: v_writelane_b32 v22, s3, 57 -; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[0:7] -; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: v_writelane_b32 v22, s0, 58 ; CHECK-NEXT: v_writelane_b32 v22, s1, 59 ; CHECK-NEXT: v_writelane_b32 v22, s2, 60 -; CHECK-NEXT: ; implicit-def: $vgpr23 : SGPR spill to VGPR lane ; CHECK-NEXT: v_writelane_b32 v22, s3, 61 -; CHECK-NEXT: v_writelane_b32 v22, s4, 62 -; CHECK-NEXT: v_writelane_b32 v23, s6, 0 -; CHECK-NEXT: v_writelane_b32 v22, s5, 63 -; CHECK-NEXT: v_writelane_b32 v23, s7, 1 +; CHECK-NEXT: ;;#ASMSTART +; CHECK-NEXT: ; def s[0:7] +; CHECK-NEXT: ;;#ASMEND +; CHECK-NEXT: ; implicit-def: $vgpr23 : SGPR spill to VGPR lane +; CHECK-NEXT: v_writelane_b32 v22, s0, 62 +; CHECK-NEXT: v_writelane_b32 v23, s2, 0 +; CHECK-NEXT: v_writelane_b32 v23, s3, 1 +; CHECK-NEXT: v_writelane_b32 v23, s4, 2 +; CHECK-NEXT: v_writelane_b32 v23, s5, 3 +; CHECK-NEXT: v_writelane_b32 v23, s6, 4 +; CHECK-NEXT: v_writelane_b32 v22, s1, 63 +; CHECK-NEXT: v_writelane_b32 v23, s7, 5 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[0:15] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v23, s0, 2 -; CHECK-NEXT: v_writelane_b32 v23, s1, 3 -; CHECK-NEXT: v_writelane_b32 v23, s2, 4 -; CHECK-NEXT: v_writelane_b32 v23, s3, 5 -; CHECK-NEXT: v_writelane_b32 v23, s4, 6 -; CHECK-NEXT: v_writelane_b32 v23, s5, 7 -; CHECK-NEXT: v_writelane_b32 v23, s6, 8 -; CHECK-NEXT: v_writelane_b32 v23, s7, 9 -; CHECK-NEXT: v_writelane_b32 v23, s8, 10 -; CHECK-NEXT: v_writelane_b32 v23, s9, 11 -; CHECK-NEXT: v_writelane_b32 v23, s10, 12 -; CHECK-NEXT: v_writelane_b32 v23, s11, 13 -; CHECK-NEXT: v_writelane_b32 v23, s12, 14 -; CHECK-NEXT: v_writelane_b32 v23, s13, 15 -; CHECK-NEXT: v_writelane_b32 v23, s14, 16 -; CHECK-NEXT: v_writelane_b32 v23, s15, 17 +; CHECK-NEXT: v_writelane_b32 v23, s0, 6 +; CHECK-NEXT: v_writelane_b32 v23, s1, 7 +; CHECK-NEXT: v_writelane_b32 v23, s2, 8 +; CHECK-NEXT: v_writelane_b32 v23, s3, 9 +; CHECK-NEXT: v_writelane_b32 v23, s4, 10 +; CHECK-NEXT: v_writelane_b32 v23, s5, 11 +; CHECK-NEXT: v_writelane_b32 v23, s6, 12 +; CHECK-NEXT: v_writelane_b32 v23, s7, 13 +; CHECK-NEXT: v_writelane_b32 v23, s8, 14 +; CHECK-NEXT: v_writelane_b32 v23, s9, 15 +; CHECK-NEXT: v_writelane_b32 v23, s10, 16 +; CHECK-NEXT: v_writelane_b32 v23, s11, 17 +; CHECK-NEXT: v_writelane_b32 v23, s12, 18 +; CHECK-NEXT: v_writelane_b32 v23, s13, 19 +; CHECK-NEXT: v_writelane_b32 v23, s14, 20 +; CHECK-NEXT: v_writelane_b32 v23, s15, 21 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[0:1] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v23, s0, 18 -; CHECK-NEXT: v_writelane_b32 v23, s1, 19 +; CHECK-NEXT: v_writelane_b32 v23, s0, 22 +; CHECK-NEXT: v_writelane_b32 v23, s1, 23 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[0:3] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v23, s0, 20 -; CHECK-NEXT: v_writelane_b32 v23, s1, 21 -; CHECK-NEXT: v_writelane_b32 v23, s2, 22 -; CHECK-NEXT: v_writelane_b32 v23, s3, 23 -; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; def s[0:7] -; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: v_writelane_b32 v23, s0, 24 ; CHECK-NEXT: v_writelane_b32 v23, s1, 25 ; CHECK-NEXT: v_writelane_b32 v23, s2, 26 ; CHECK-NEXT: v_writelane_b32 v23, s3, 27 -; CHECK-NEXT: v_writelane_b32 v23, s4, 28 -; CHECK-NEXT: v_writelane_b32 v23, s5, 29 -; CHECK-NEXT: v_writelane_b32 v23, s6, 30 -; CHECK-NEXT: v_writelane_b32 v23, s7, 31 +; CHECK-NEXT: ;;#ASMSTART +; CHECK-NEXT: ; def s[0:7] +; CHECK-NEXT: ;;#ASMEND +; CHECK-NEXT: v_writelane_b32 v23, s0, 28 +; CHECK-NEXT: v_writelane_b32 v23, s1, 29 +; CHECK-NEXT: v_writelane_b32 v23, s2, 30 +; CHECK-NEXT: v_writelane_b32 v23, s3, 31 +; CHECK-NEXT: v_writelane_b32 v23, s4, 32 +; CHECK-NEXT: v_writelane_b32 v23, s5, 33 +; CHECK-NEXT: v_writelane_b32 v23, s6, 34 +; CHECK-NEXT: v_writelane_b32 v23, s7, 35 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; def s[0:15] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_writelane_b32 v23, s0, 32 -; CHECK-NEXT: v_writelane_b32 v23, s1, 33 -; CHECK-NEXT: v_writelane_b32 v23, s2, 34 -; CHECK-NEXT: v_writelane_b32 v23, s3, 35 -; CHECK-NEXT: v_writelane_b32 v23, s4, 36 -; CHECK-NEXT: v_writelane_b32 v23, s5, 37 -; CHECK-NEXT: v_writelane_b32 v23, s6, 38 -; CHECK-NEXT: v_writelane_b32 v23, s7, 39 -; CHECK-NEXT: v_writelane_b32 v23, s8, 40 -; CHECK-NEXT: v_writelane_b32 v23, s9, 41 -; CHECK-NEXT: v_writelane_b32 v23, s10, 42 -; CHECK-NEXT: v_writelane_b32 v23, s11, 43 -; CHECK-NEXT: v_writelane_b32 v23, s12, 44 -; CHECK-NEXT: v_writelane_b32 v23, s13, 45 -; CHECK-NEXT: v_writelane_b32 v23, s14, 46 -; CHECK-NEXT: v_writelane_b32 v23, s15, 47 +; CHECK-NEXT: v_writelane_b32 v23, s0, 36 +; CHECK-NEXT: v_writelane_b32 v23, s1, 37 +; CHECK-NEXT: v_writelane_b32 v23, s2, 38 +; CHECK-NEXT: v_writelane_b32 v23, s3, 39 +; CHECK-NEXT: v_writelane_b32 v23, s4, 40 +; CHECK-NEXT: v_writelane_b32 v23, s5, 41 +; CHECK-NEXT: v_writelane_b32 v23, s6, 42 +; CHECK-NEXT: v_writelane_b32 v23, s7, 43 +; CHECK-NEXT: v_writelane_b32 v23, s8, 44 +; CHECK-NEXT: v_writelane_b32 v23, s9, 45 +; CHECK-NEXT: v_writelane_b32 v23, s10, 46 +; CHECK-NEXT: v_writelane_b32 v23, s11, 47 +; CHECK-NEXT: v_writelane_b32 v23, s12, 48 +; CHECK-NEXT: v_writelane_b32 v23, s13, 49 +; CHECK-NEXT: v_writelane_b32 v23, s14, 50 +; CHECK-NEXT: v_writelane_b32 v23, s15, 51 ; CHECK-NEXT: s_cbranch_scc0 .LBB0_2 ; CHECK-NEXT: ; %bb.1: ; %ret ; CHECK-NEXT: s_endpgm @@ -206,166 +210,170 @@ define amdgpu_kernel void @kernel0(ptr addrspace(1) %out, i32 %in) #1 { ; CHECK-NEXT: v_readlane_b32 s1, v22, 3 ; CHECK-NEXT: v_readlane_b32 s2, v22, 4 ; CHECK-NEXT: v_readlane_b32 s3, v22, 5 +; CHECK-NEXT: v_readlane_b32 s4, v22, 6 +; CHECK-NEXT: v_readlane_b32 s5, v22, 7 +; CHECK-NEXT: v_readlane_b32 s6, v22, 8 +; CHECK-NEXT: v_readlane_b32 s7, v22, 9 ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; use s[0:3] +; CHECK-NEXT: ; use s[48:51] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v22, 6 -; CHECK-NEXT: v_readlane_b32 s1, v22, 7 -; CHECK-NEXT: v_readlane_b32 s2, v22, 8 -; CHECK-NEXT: v_readlane_b32 s3, v22, 9 -; CHECK-NEXT: v_readlane_b32 s4, v22, 10 -; CHECK-NEXT: v_readlane_b32 s5, v22, 11 -; CHECK-NEXT: v_readlane_b32 s6, v22, 12 -; CHECK-NEXT: v_readlane_b32 s7, v22, 13 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:7] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v22, 14 -; CHECK-NEXT: v_readlane_b32 s1, v22, 15 -; CHECK-NEXT: v_readlane_b32 s2, v22, 16 -; CHECK-NEXT: v_readlane_b32 s3, v22, 17 -; CHECK-NEXT: v_readlane_b32 s4, v22, 18 -; CHECK-NEXT: v_readlane_b32 s5, v22, 19 -; CHECK-NEXT: v_readlane_b32 s6, v22, 20 -; CHECK-NEXT: v_readlane_b32 s7, v22, 21 -; CHECK-NEXT: v_readlane_b32 s8, v22, 22 -; CHECK-NEXT: v_readlane_b32 s9, v22, 23 -; CHECK-NEXT: v_readlane_b32 s10, v22, 24 -; CHECK-NEXT: v_readlane_b32 s11, v22, 25 -; CHECK-NEXT: v_readlane_b32 s12, v22, 26 -; CHECK-NEXT: v_readlane_b32 s13, v22, 27 -; CHECK-NEXT: v_readlane_b32 s14, v22, 28 -; CHECK-NEXT: v_readlane_b32 s15, v22, 29 +; CHECK-NEXT: v_readlane_b32 s0, v22, 10 +; CHECK-NEXT: v_readlane_b32 s1, v22, 11 +; CHECK-NEXT: v_readlane_b32 s2, v22, 12 +; CHECK-NEXT: v_readlane_b32 s3, v22, 13 +; CHECK-NEXT: v_readlane_b32 s4, v22, 14 +; CHECK-NEXT: v_readlane_b32 s5, v22, 15 +; CHECK-NEXT: v_readlane_b32 s6, v22, 16 +; CHECK-NEXT: v_readlane_b32 s7, v22, 17 +; CHECK-NEXT: v_readlane_b32 s8, v22, 18 +; CHECK-NEXT: v_readlane_b32 s9, v22, 19 +; CHECK-NEXT: v_readlane_b32 s10, v22, 20 +; CHECK-NEXT: v_readlane_b32 s11, v22, 21 +; CHECK-NEXT: v_readlane_b32 s12, v22, 22 +; CHECK-NEXT: v_readlane_b32 s13, v22, 23 +; CHECK-NEXT: v_readlane_b32 s14, v22, 24 +; CHECK-NEXT: v_readlane_b32 s15, v22, 25 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:15] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v22, 30 -; CHECK-NEXT: v_readlane_b32 s1, v22, 31 -; CHECK-NEXT: v_readlane_b32 s2, v22, 32 -; CHECK-NEXT: v_readlane_b32 s3, v22, 33 -; CHECK-NEXT: v_readlane_b32 s4, v22, 34 -; CHECK-NEXT: v_readlane_b32 s5, v22, 35 -; CHECK-NEXT: v_readlane_b32 s6, v22, 36 -; CHECK-NEXT: v_readlane_b32 s7, v22, 37 +; CHECK-NEXT: v_readlane_b32 s0, v22, 26 +; CHECK-NEXT: v_readlane_b32 s1, v22, 27 +; CHECK-NEXT: v_readlane_b32 s2, v22, 28 +; CHECK-NEXT: v_readlane_b32 s3, v22, 29 +; CHECK-NEXT: v_readlane_b32 s4, v22, 30 +; CHECK-NEXT: v_readlane_b32 s5, v22, 31 +; CHECK-NEXT: v_readlane_b32 s6, v22, 32 +; CHECK-NEXT: v_readlane_b32 s7, v22, 33 ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; use s[42:43] +; CHECK-NEXT: ; use s[38:39] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; use s[52:55] +; CHECK-NEXT: ; use s[44:47] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:7] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v22, 38 -; CHECK-NEXT: v_readlane_b32 s1, v22, 39 -; CHECK-NEXT: v_readlane_b32 s2, v22, 40 -; CHECK-NEXT: v_readlane_b32 s3, v22, 41 +; CHECK-NEXT: v_readlane_b32 s0, v22, 34 +; CHECK-NEXT: v_readlane_b32 s1, v22, 35 +; CHECK-NEXT: v_readlane_b32 s2, v22, 36 +; CHECK-NEXT: v_readlane_b32 s3, v22, 37 +; CHECK-NEXT: v_readlane_b32 s4, v22, 38 +; CHECK-NEXT: v_readlane_b32 s5, v22, 39 +; CHECK-NEXT: v_readlane_b32 s6, v22, 40 +; CHECK-NEXT: v_readlane_b32 s7, v22, 41 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[16:31] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; use s[40:41] +; CHECK-NEXT: ; use s[36:37] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; use s[36:39] +; CHECK-NEXT: ; use s[40:43] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; use s[44:51] +; CHECK-NEXT: ; use s[0:7] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s4, v22, 42 -; CHECK-NEXT: v_readlane_b32 s5, v22, 43 -; CHECK-NEXT: v_readlane_b32 s6, v22, 44 -; CHECK-NEXT: v_readlane_b32 s7, v22, 45 -; CHECK-NEXT: v_readlane_b32 s8, v22, 46 -; CHECK-NEXT: v_readlane_b32 s9, v22, 47 -; CHECK-NEXT: v_readlane_b32 s10, v22, 48 -; CHECK-NEXT: v_readlane_b32 s11, v22, 49 -; CHECK-NEXT: v_readlane_b32 s12, v22, 50 -; CHECK-NEXT: v_readlane_b32 s13, v22, 51 -; CHECK-NEXT: v_readlane_b32 s14, v22, 52 -; CHECK-NEXT: v_readlane_b32 s15, v22, 53 +; CHECK-NEXT: v_readlane_b32 s0, v22, 42 +; CHECK-NEXT: v_readlane_b32 s1, v22, 43 +; CHECK-NEXT: v_readlane_b32 s2, v22, 44 +; CHECK-NEXT: v_readlane_b32 s3, v22, 45 +; CHECK-NEXT: v_readlane_b32 s4, v22, 46 +; CHECK-NEXT: v_readlane_b32 s5, v22, 47 +; CHECK-NEXT: v_readlane_b32 s6, v22, 48 +; CHECK-NEXT: v_readlane_b32 s7, v22, 49 +; CHECK-NEXT: v_readlane_b32 s8, v22, 50 +; CHECK-NEXT: v_readlane_b32 s9, v22, 51 +; CHECK-NEXT: v_readlane_b32 s10, v22, 52 +; CHECK-NEXT: v_readlane_b32 s11, v22, 53 +; CHECK-NEXT: v_readlane_b32 s12, v22, 54 +; CHECK-NEXT: v_readlane_b32 s13, v22, 55 +; CHECK-NEXT: v_readlane_b32 s14, v22, 56 +; CHECK-NEXT: v_readlane_b32 s15, v22, 57 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:15] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v22, 54 -; CHECK-NEXT: v_readlane_b32 s1, v22, 55 -; CHECK-NEXT: v_readlane_b32 s2, v22, 56 -; CHECK-NEXT: v_readlane_b32 s3, v22, 57 +; CHECK-NEXT: v_readlane_b32 s0, v22, 58 +; CHECK-NEXT: v_readlane_b32 s1, v22, 59 +; CHECK-NEXT: v_readlane_b32 s2, v22, 60 +; CHECK-NEXT: v_readlane_b32 s3, v22, 61 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[34:35] ; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:3] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v22, 58 -; CHECK-NEXT: v_readlane_b32 s1, v22, 59 -; CHECK-NEXT: v_readlane_b32 s2, v22, 60 -; CHECK-NEXT: v_readlane_b32 s3, v22, 61 -; CHECK-NEXT: v_readlane_b32 s4, v22, 62 -; CHECK-NEXT: v_readlane_b32 s5, v22, 63 -; CHECK-NEXT: v_readlane_b32 s6, v23, 0 -; CHECK-NEXT: v_readlane_b32 s7, v23, 1 +; CHECK-NEXT: v_readlane_b32 s0, v22, 62 +; CHECK-NEXT: v_readlane_b32 s1, v22, 63 +; CHECK-NEXT: v_readlane_b32 s2, v23, 0 +; CHECK-NEXT: v_readlane_b32 s3, v23, 1 +; CHECK-NEXT: v_readlane_b32 s4, v23, 2 +; CHECK-NEXT: v_readlane_b32 s5, v23, 3 +; CHECK-NEXT: v_readlane_b32 s6, v23, 4 +; CHECK-NEXT: v_readlane_b32 s7, v23, 5 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:7] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v23, 2 -; CHECK-NEXT: v_readlane_b32 s1, v23, 3 -; CHECK-NEXT: v_readlane_b32 s2, v23, 4 -; CHECK-NEXT: v_readlane_b32 s3, v23, 5 -; CHECK-NEXT: v_readlane_b32 s4, v23, 6 -; CHECK-NEXT: v_readlane_b32 s5, v23, 7 -; CHECK-NEXT: v_readlane_b32 s6, v23, 8 -; CHECK-NEXT: v_readlane_b32 s7, v23, 9 -; CHECK-NEXT: v_readlane_b32 s8, v23, 10 -; CHECK-NEXT: v_readlane_b32 s9, v23, 11 -; CHECK-NEXT: v_readlane_b32 s10, v23, 12 -; CHECK-NEXT: v_readlane_b32 s11, v23, 13 -; CHECK-NEXT: v_readlane_b32 s12, v23, 14 -; CHECK-NEXT: v_readlane_b32 s13, v23, 15 -; CHECK-NEXT: v_readlane_b32 s14, v23, 16 -; CHECK-NEXT: v_readlane_b32 s15, v23, 17 +; CHECK-NEXT: v_readlane_b32 s0, v23, 6 +; CHECK-NEXT: v_readlane_b32 s1, v23, 7 +; CHECK-NEXT: v_readlane_b32 s2, v23, 8 +; CHECK-NEXT: v_readlane_b32 s3, v23, 9 +; CHECK-NEXT: v_readlane_b32 s4, v23, 10 +; CHECK-NEXT: v_readlane_b32 s5, v23, 11 +; CHECK-NEXT: v_readlane_b32 s6, v23, 12 +; CHECK-NEXT: v_readlane_b32 s7, v23, 13 +; CHECK-NEXT: v_readlane_b32 s8, v23, 14 +; CHECK-NEXT: v_readlane_b32 s9, v23, 15 +; CHECK-NEXT: v_readlane_b32 s10, v23, 16 +; CHECK-NEXT: v_readlane_b32 s11, v23, 17 +; CHECK-NEXT: v_readlane_b32 s12, v23, 18 +; CHECK-NEXT: v_readlane_b32 s13, v23, 19 +; CHECK-NEXT: v_readlane_b32 s14, v23, 20 +; CHECK-NEXT: v_readlane_b32 s15, v23, 21 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:15] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v23, 18 -; CHECK-NEXT: v_readlane_b32 s1, v23, 19 +; CHECK-NEXT: v_readlane_b32 s0, v23, 22 +; CHECK-NEXT: v_readlane_b32 s1, v23, 23 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:1] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v23, 20 -; CHECK-NEXT: v_readlane_b32 s1, v23, 21 -; CHECK-NEXT: v_readlane_b32 s2, v23, 22 -; CHECK-NEXT: v_readlane_b32 s3, v23, 23 -; CHECK-NEXT: ;;#ASMSTART -; CHECK-NEXT: ; use s[0:3] -; CHECK-NEXT: ;;#ASMEND ; CHECK-NEXT: v_readlane_b32 s0, v23, 24 ; CHECK-NEXT: v_readlane_b32 s1, v23, 25 ; CHECK-NEXT: v_readlane_b32 s2, v23, 26 ; CHECK-NEXT: v_readlane_b32 s3, v23, 27 -; CHECK-NEXT: v_readlane_b32 s4, v23, 28 -; CHECK-NEXT: v_readlane_b32 s5, v23, 29 -; CHECK-NEXT: v_readlane_b32 s6, v23, 30 -; CHECK-NEXT: v_readlane_b32 s7, v23, 31 +; CHECK-NEXT: ;;#ASMSTART +; CHECK-NEXT: ; use s[0:3] +; CHECK-NEXT: ;;#ASMEND +; CHECK-NEXT: v_readlane_b32 s0, v23, 28 +; CHECK-NEXT: v_readlane_b32 s1, v23, 29 +; CHECK-NEXT: v_readlane_b32 s2, v23, 30 +; CHECK-NEXT: v_readlane_b32 s3, v23, 31 +; CHECK-NEXT: v_readlane_b32 s4, v23, 32 +; CHECK-NEXT: v_readlane_b32 s5, v23, 33 +; CHECK-NEXT: v_readlane_b32 s6, v23, 34 +; CHECK-NEXT: v_readlane_b32 s7, v23, 35 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:7] ; CHECK-NEXT: ;;#ASMEND -; CHECK-NEXT: v_readlane_b32 s0, v23, 32 -; CHECK-NEXT: v_readlane_b32 s1, v23, 33 -; CHECK-NEXT: v_readlane_b32 s2, v23, 34 -; CHECK-NEXT: v_readlane_b32 s3, v23, 35 -; CHECK-NEXT: v_readlane_b32 s4, v23, 36 -; CHECK-NEXT: v_readlane_b32 s5, v23, 37 -; CHECK-NEXT: v_readlane_b32 s6, v23, 38 -; CHECK-NEXT: v_readlane_b32 s7, v23, 39 -; CHECK-NEXT: v_readlane_b32 s8, v23, 40 -; CHECK-NEXT: v_readlane_b32 s9, v23, 41 -; CHECK-NEXT: v_readlane_b32 s10, v23, 42 -; CHECK-NEXT: v_readlane_b32 s11, v23, 43 -; CHECK-NEXT: v_readlane_b32 s12, v23, 44 -; CHECK-NEXT: v_readlane_b32 s13, v23, 45 -; CHECK-NEXT: v_readlane_b32 s14, v23, 46 -; CHECK-NEXT: v_readlane_b32 s15, v23, 47 +; CHECK-NEXT: v_readlane_b32 s0, v23, 36 +; CHECK-NEXT: v_readlane_b32 s1, v23, 37 +; CHECK-NEXT: v_readlane_b32 s2, v23, 38 +; CHECK-NEXT: v_readlane_b32 s3, v23, 39 +; CHECK-NEXT: v_readlane_b32 s4, v23, 40 +; CHECK-NEXT: v_readlane_b32 s5, v23, 41 +; CHECK-NEXT: v_readlane_b32 s6, v23, 42 +; CHECK-NEXT: v_readlane_b32 s7, v23, 43 +; CHECK-NEXT: v_readlane_b32 s8, v23, 44 +; CHECK-NEXT: v_readlane_b32 s9, v23, 45 +; CHECK-NEXT: v_readlane_b32 s10, v23, 46 +; CHECK-NEXT: v_readlane_b32 s11, v23, 47 +; CHECK-NEXT: v_readlane_b32 s12, v23, 48 +; CHECK-NEXT: v_readlane_b32 s13, v23, 49 +; CHECK-NEXT: v_readlane_b32 s14, v23, 50 +; CHECK-NEXT: v_readlane_b32 s15, v23, 51 ; CHECK-NEXT: ;;#ASMSTART ; CHECK-NEXT: ; use s[0:15] ; CHECK-NEXT: ;;#ASMEND diff --git a/llvm/test/CodeGen/AMDGPU/sgpr-spill-no-vgprs.ll b/llvm/test/CodeGen/AMDGPU/sgpr-spill-no-vgprs.ll index 455d22f2aa29c..cdfba3cf0db7f 100644 --- a/llvm/test/CodeGen/AMDGPU/sgpr-spill-no-vgprs.ll +++ b/llvm/test/CodeGen/AMDGPU/sgpr-spill-no-vgprs.ll @@ -7,7 +7,7 @@ define amdgpu_kernel void @partial_no_vgprs_last_sgpr_spill(ptr addrspace(1) %out, i32 %in) #1 { ; GCN-LABEL: partial_no_vgprs_last_sgpr_spill: ; GCN: ; %bb.0: -; GCN-NEXT: s_add_u32 s0, s0, s15 +; GCN-NEXT: s_add_u32 s0, s0, s17 ; GCN-NEXT: s_addc_u32 s1, s1, 0 ; GCN-NEXT: s_load_dword s4, s[8:9], 0x2 ; GCN-NEXT: ;;#ASMSTART diff --git a/llvm/test/CodeGen/AMDGPU/shift-i128.ll b/llvm/test/CodeGen/AMDGPU/shift-i128.ll index a423b6f831a9d..65a17ed67481c 100644 --- a/llvm/test/CodeGen/AMDGPU/shift-i128.ll +++ b/llvm/test/CodeGen/AMDGPU/shift-i128.ll @@ -182,8 +182,10 @@ define amdgpu_kernel void @s_shl_i128_ss(i128 %lhs, i128 %rhs) { ; GCN-LABEL: s_shl_i128_ss: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: v_mov_b32_e32 v4, 0 -; GCN-NEXT: v_mov_b32_e32 v5, 0 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_sub_i32 s5, s4, 64 ; GCN-NEXT: s_sub_i32 s12, 64, s4 @@ -203,6 +205,7 @@ define amdgpu_kernel void @s_shl_i128_ss(i128 %lhs, i128 %rhs) { ; GCN-NEXT: v_mov_b32_e32 v0, s6 ; GCN-NEXT: v_mov_b32_e32 v1, s5 ; GCN-NEXT: v_mov_b32_e32 v2, s0 +; GCN-NEXT: v_mov_b32_e32 v5, 0 ; GCN-NEXT: v_mov_b32_e32 v3, s1 ; GCN-NEXT: flat_store_dwordx4 v[4:5], v[0:3] ; GCN-NEXT: s_endpgm @@ -215,8 +218,10 @@ define amdgpu_kernel void @s_lshr_i128_ss(i128 %lhs, i128 %rhs) { ; GCN-LABEL: s_lshr_i128_ss: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: v_mov_b32_e32 v4, 0 -; GCN-NEXT: v_mov_b32_e32 v5, 0 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_sub_i32 s5, s4, 64 ; GCN-NEXT: s_sub_i32 s12, 64, s4 @@ -236,6 +241,7 @@ define amdgpu_kernel void @s_lshr_i128_ss(i128 %lhs, i128 %rhs) { ; GCN-NEXT: v_mov_b32_e32 v0, s0 ; GCN-NEXT: v_mov_b32_e32 v1, s1 ; GCN-NEXT: v_mov_b32_e32 v2, s6 +; GCN-NEXT: v_mov_b32_e32 v5, 0 ; GCN-NEXT: v_mov_b32_e32 v3, s5 ; GCN-NEXT: flat_store_dwordx4 v[4:5], v[0:3] ; GCN-NEXT: s_endpgm @@ -248,8 +254,10 @@ define amdgpu_kernel void @s_ashr_i128_ss(i128 %lhs, i128 %rhs) { ; GCN-LABEL: s_ashr_i128_ss: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: v_mov_b32_e32 v4, 0 -; GCN-NEXT: v_mov_b32_e32 v5, 0 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_sub_i32 s5, 64, s4 ; GCN-NEXT: s_lshr_b64 s[6:7], s[0:1], s4 @@ -270,6 +278,7 @@ define amdgpu_kernel void @s_ashr_i128_ss(i128 %lhs, i128 %rhs) { ; GCN-NEXT: v_mov_b32_e32 v0, s0 ; GCN-NEXT: v_mov_b32_e32 v1, s1 ; GCN-NEXT: v_mov_b32_e32 v2, s2 +; GCN-NEXT: v_mov_b32_e32 v5, 0 ; GCN-NEXT: v_mov_b32_e32 v3, s3 ; GCN-NEXT: flat_store_dwordx4 v[4:5], v[0:3] ; GCN-NEXT: s_endpgm @@ -430,6 +439,9 @@ define <2 x i128> @v_ashr_v2i128_vv(<2 x i128> %lhs, <2 x i128> %rhs) { define amdgpu_kernel void @s_shl_v2i128ss(<2 x i128> %lhs, <2 x i128> %rhs) { ; GCN-LABEL: s_shl_v2i128ss: ; GCN: ; %bb.0: +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_load_dwordx16 s[0:15], s[8:9], 0x0 ; GCN-NEXT: v_mov_b32_e32 v6, 16 ; GCN-NEXT: v_mov_b32_e32 v4, 0 @@ -502,6 +514,9 @@ define amdgpu_kernel void @s_shl_v2i128ss(<2 x i128> %lhs, <2 x i128> %rhs) { define amdgpu_kernel void @s_lshr_v2i128_ss(<2 x i128> %lhs, <2 x i128> %rhs) { ; GCN-LABEL: s_lshr_v2i128_ss: ; GCN: ; %bb.0: +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_load_dwordx16 s[0:15], s[8:9], 0x0 ; GCN-NEXT: v_mov_b32_e32 v6, 16 ; GCN-NEXT: v_mov_b32_e32 v4, 0 @@ -574,6 +589,9 @@ define amdgpu_kernel void @s_lshr_v2i128_ss(<2 x i128> %lhs, <2 x i128> %rhs) { define amdgpu_kernel void @s_ashr_v2i128_ss(<2 x i128> %lhs, <2 x i128> %rhs) { ; GCN-LABEL: s_ashr_v2i128_ss: ; GCN: ; %bb.0: +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_load_dwordx16 s[0:15], s[8:9], 0x0 ; GCN-NEXT: v_mov_b32_e32 v6, 16 ; GCN-NEXT: v_mov_b32_e32 v4, 0 diff --git a/llvm/test/CodeGen/AMDGPU/simple-indirect-call.ll b/llvm/test/CodeGen/AMDGPU/simple-indirect-call.ll index 8531b2ad4e405..3c47e2504747d 100644 --- a/llvm/test/CodeGen/AMDGPU/simple-indirect-call.ll +++ b/llvm/test/CodeGen/AMDGPU/simple-indirect-call.ll @@ -1,5 +1,4 @@ ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --check-globals -; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -amdgpu-annotate-kernel-features %s | FileCheck -check-prefix=AKF_GCN %s ; RUN: opt -S -mtriple=amdgcn-amd-amdhsa -passes=amdgpu-attributor %s | FileCheck -check-prefix=ATTRIBUTOR_GCN %s ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 < %s | FileCheck -check-prefix=GFX9 %s @@ -7,9 +6,6 @@ target datalayout = "A5" define internal void @indirect() { -; AKF_GCN-LABEL: define {{[^@]+}}@indirect() { -; AKF_GCN-NEXT: ret void -; ; ATTRIBUTOR_GCN-LABEL: define {{[^@]+}}@indirect ; ATTRIBUTOR_GCN-SAME: () #[[ATTR0:[0-9]+]] { ; ATTRIBUTOR_GCN-NEXT: ret void @@ -22,15 +18,6 @@ define internal void @indirect() { } define amdgpu_kernel void @test_simple_indirect_call() { -; AKF_GCN-LABEL: define {{[^@]+}}@test_simple_indirect_call -; AKF_GCN-SAME: () #[[ATTR0:[0-9]+]] { -; AKF_GCN-NEXT: [[FPTR:%.*]] = alloca ptr, align 8, addrspace(5) -; AKF_GCN-NEXT: [[FPTR_CAST:%.*]] = addrspacecast ptr addrspace(5) [[FPTR]] to ptr -; AKF_GCN-NEXT: store ptr @indirect, ptr [[FPTR_CAST]], align 8 -; AKF_GCN-NEXT: [[FP:%.*]] = load ptr, ptr [[FPTR_CAST]], align 8 -; AKF_GCN-NEXT: call void [[FP]]() -; AKF_GCN-NEXT: ret void -; ; ATTRIBUTOR_GCN-LABEL: define {{[^@]+}}@test_simple_indirect_call ; ATTRIBUTOR_GCN-SAME: () #[[ATTR1:[0-9]+]] { ; ATTRIBUTOR_GCN-NEXT: [[FPTR:%.*]] = alloca ptr, align 8, addrspace(5) @@ -79,12 +66,10 @@ define amdgpu_kernel void @test_simple_indirect_call() { !llvm.module.flags = !{!0} !0 = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. -; AKF_GCN: attributes #[[ATTR0]] = { "amdgpu-calls" "amdgpu-stack-objects" } ;. ; ATTRIBUTOR_GCN: attributes #[[ATTR0]] = { "amdgpu-agpr-alloc"="0" "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } ; ATTRIBUTOR_GCN: attributes #[[ATTR1]] = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } ;. -; AKF_GCN: [[META0:![0-9]+]] = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. ; ATTRIBUTOR_GCN: [[META0:![0-9]+]] = !{i32 1, !"amdhsa_code_object_version", i32 500} ;. diff --git a/llvm/test/CodeGen/AMDGPU/sint_to_fp.f64.ll b/llvm/test/CodeGen/AMDGPU/sint_to_fp.f64.ll index 8129a7ac51df9..d71d0f78fe1c3 100644 --- a/llvm/test/CodeGen/AMDGPU/sint_to_fp.f64.ll +++ b/llvm/test/CodeGen/AMDGPU/sint_to_fp.f64.ll @@ -9,6 +9,9 @@ define amdgpu_kernel void @sint_to_fp_i32_to_f64(ptr addrspace(1) %out, i32 %in) ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f64_i32_e32 v[0:1], s2 ; CI-NEXT: v_mov_b32_e32 v3, s1 @@ -20,6 +23,9 @@ define amdgpu_kernel void @sint_to_fp_i32_to_f64(ptr addrspace(1) %out, i32 %in) ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f64_i32_e32 v[0:1], s2 ; VI-NEXT: v_mov_b32_e32 v3, s1 @@ -38,11 +44,14 @@ define amdgpu_kernel void @sint_to_fp_i1_f64(ptr addrspace(1) %out, i32 %in) { ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; CI-NEXT: v_mov_b32_e32 v0, 0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_cmp_eq_u32 s2, 0 ; CI-NEXT: s_cselect_b32 s2, 0xbff00000, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v0, 0 ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -52,11 +61,14 @@ define amdgpu_kernel void @sint_to_fp_i1_f64(ptr addrspace(1) %out, i32 %in) { ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0xbff00000, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -72,6 +84,9 @@ define amdgpu_kernel void @sint_to_fp_i1_f64_load(ptr addrspace(1) %out, i1 %in) ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_bitcmp1_b32 s2, 0 ; CI-NEXT: s_cselect_b64 s[2:3], -1, 0 @@ -86,6 +101,9 @@ define amdgpu_kernel void @sint_to_fp_i1_f64_load(ptr addrspace(1) %out, i1 %in) ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_bitcmp1_b32 s2, 0 ; VI-NEXT: s_cselect_b64 s[2:3], -1, 0 @@ -104,6 +122,9 @@ define amdgpu_kernel void @s_sint_to_fp_i64_to_f64(ptr addrspace(1) %out, i64 %i ; CI-LABEL: s_sint_to_fp_i64_to_f64: ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_cvt_f64_i32_e32 v[0:1], s3 ; CI-NEXT: v_cvt_f64_u32_e32 v[2:3], s2 @@ -117,6 +138,9 @@ define amdgpu_kernel void @s_sint_to_fp_i64_to_f64(ptr addrspace(1) %out, i64 %i ; VI-LABEL: s_sint_to_fp_i64_to_f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f64_i32_e32 v[0:1], s3 ; VI-NEXT: v_cvt_f64_u32_e32 v[2:3], s2 @@ -136,6 +160,9 @@ define amdgpu_kernel void @v_sint_to_fp_i64_to_f64(ptr addrspace(1) %out, ptr ad ; CI: ; %bb.0: ; CI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; CI-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: v_mov_b32_e32 v1, s3 ; CI-NEXT: v_add_i32_e32 v0, vcc, s2, v0 @@ -155,6 +182,9 @@ define amdgpu_kernel void @v_sint_to_fp_i64_to_f64(ptr addrspace(1) %out, ptr ad ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -183,6 +213,9 @@ define amdgpu_kernel void @s_sint_to_fp_i8_to_f64(ptr addrspace(1) %out, i8 %in) ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_sext_i32_i8 s2, s2 ; CI-NEXT: v_cvt_f64_i32_e32 v[0:1], s2 @@ -195,6 +228,9 @@ define amdgpu_kernel void @s_sint_to_fp_i8_to_f64(ptr addrspace(1) %out, i8 %in) ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_sext_i32_i8 s2, s2 ; VI-NEXT: v_cvt_f64_i32_e32 v[0:1], s2 @@ -231,11 +267,14 @@ define amdgpu_kernel void @s_select_sint_to_fp_i1_vals_f64(ptr addrspace(1) %out ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; CI-NEXT: v_mov_b32_e32 v0, 0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_cmp_eq_u32 s2, 0 ; CI-NEXT: s_cselect_b32 s2, 0xbff00000, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v0, 0 ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -245,11 +284,14 @@ define amdgpu_kernel void @s_select_sint_to_fp_i1_vals_f64(ptr addrspace(1) %out ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0xbff00000, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -282,11 +324,14 @@ define amdgpu_kernel void @s_select_sint_to_fp_i1_vals_i64(ptr addrspace(1) %out ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; CI-NEXT: v_mov_b32_e32 v0, 0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_cmp_eq_u32 s2, 0 ; CI-NEXT: s_cselect_b32 s2, 0xbff00000, 0 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v0, 0 ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -296,11 +341,14 @@ define amdgpu_kernel void @s_select_sint_to_fp_i1_vals_i64(ptr addrspace(1) %out ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0xbff00000, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -352,11 +400,14 @@ define amdgpu_kernel void @s_swap_select_sint_to_fp_i1_vals_f64(ptr addrspace(1) ; CI: ; %bb.0: ; CI-NEXT: s_load_dword s2, s[8:9], 0x2 ; CI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; CI-NEXT: v_mov_b32_e32 v0, 0 +; CI-NEXT: s_add_i32 s12, s12, s17 +; CI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; CI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; CI-NEXT: s_waitcnt lgkmcnt(0) ; CI-NEXT: s_cmp_eq_u32 s2, 0 ; CI-NEXT: s_cselect_b32 s2, 0, 0xbff00000 ; CI-NEXT: v_mov_b32_e32 v3, s1 +; CI-NEXT: v_mov_b32_e32 v0, 0 ; CI-NEXT: v_mov_b32_e32 v1, s2 ; CI-NEXT: v_mov_b32_e32 v2, s0 ; CI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -366,11 +417,14 @@ define amdgpu_kernel void @s_swap_select_sint_to_fp_i1_vals_f64(ptr addrspace(1) ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0, 0xbff00000 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] diff --git a/llvm/test/CodeGen/AMDGPU/spill-vector-superclass.ll b/llvm/test/CodeGen/AMDGPU/spill-vector-superclass.ll index 5ae339454a0ba..bd255e88b9512 100644 --- a/llvm/test/CodeGen/AMDGPU/spill-vector-superclass.ll +++ b/llvm/test/CodeGen/AMDGPU/spill-vector-superclass.ll @@ -12,10 +12,10 @@ define amdgpu_kernel void @test_spill_av_class(<4 x i32> %arg) #0 { ; GCN-NEXT: [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 1, implicit $exec ; GCN-NEXT: [[V_MOV_B32_e32_1:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 2, implicit $exec ; GCN-NEXT: [[V_MFMA_I32_4X4X4I8_e64_:%[0-9]+]]:areg_128 = V_MFMA_I32_4X4X4I8_e64 [[V_MOV_B32_e32_]], [[V_MOV_B32_e32_1]], [[COPY]], 0, 0, 0, implicit $mode, implicit $exec - ; GCN-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 2228234 /* regdef:VGPR_32 */, def undef %13.sub0 + ; GCN-NEXT: INLINEASM &"; def $0", 1 /* sideeffect attdialect */, 2228234 /* regdef:VGPR_32 */, def undef %14.sub0 ; GCN-NEXT: [[COPY1:%[0-9]+]]:vreg_128 = COPY [[V_MFMA_I32_4X4X4I8_e64_]] - ; GCN-NEXT: GLOBAL_STORE_DWORDX4 undef %23:vreg_64, [[COPY1]], 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) - ; GCN-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 3538953 /* reguse:VReg_64 */, %13 + ; GCN-NEXT: GLOBAL_STORE_DWORDX4 undef %24:vreg_64, [[COPY1]], 0, 0, implicit $exec :: (volatile store (s128) into `ptr addrspace(1) poison`, addrspace 1) + ; GCN-NEXT: INLINEASM &"; use $0", 1 /* sideeffect attdialect */, 3538953 /* reguse:VReg_64 */, %14 ; GCN-NEXT: S_ENDPGM 0 %v0 = call i32 asm sideeffect "; def $0", "=v"() %tmp = insertelement <2 x i32> poison, i32 %v0, i32 0 diff --git a/llvm/test/CodeGen/AMDGPU/store-weird-sizes.ll b/llvm/test/CodeGen/AMDGPU/store-weird-sizes.ll index f791135d45e9a..ef92cf3214e7f 100644 --- a/llvm/test/CodeGen/AMDGPU/store-weird-sizes.ll +++ b/llvm/test/CodeGen/AMDGPU/store-weird-sizes.ll @@ -50,7 +50,10 @@ define void @local_store_i56(ptr addrspace(3) %ptr, i56 %arg) #0 { define amdgpu_kernel void @local_store_i55(ptr addrspace(3) %ptr, i55 %arg) #0 { ; HAWAII-LABEL: local_store_i55: ; HAWAII: ; %bb.0: +; HAWAII-NEXT: s_add_i32 s12, s12, s17 ; HAWAII-NEXT: s_or_b32 s0, s8, 14 +; HAWAII-NEXT: s_mov_b32 flat_scratch_lo, s13 +; HAWAII-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; HAWAII-NEXT: v_mov_b32_e32 v0, s0 ; HAWAII-NEXT: v_mov_b32_e32 v1, s9 ; HAWAII-NEXT: flat_load_ubyte v0, v[0:1] @@ -70,7 +73,10 @@ define amdgpu_kernel void @local_store_i55(ptr addrspace(3) %ptr, i55 %arg) #0 { ; ; FIJI-LABEL: local_store_i55: ; FIJI: ; %bb.0: +; FIJI-NEXT: s_add_i32 s12, s12, s17 ; FIJI-NEXT: s_or_b32 s0, s8, 14 +; FIJI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; FIJI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; FIJI-NEXT: v_mov_b32_e32 v0, s0 ; FIJI-NEXT: v_mov_b32_e32 v1, s9 ; FIJI-NEXT: flat_load_ubyte v0, v[0:1] diff --git a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll index 19d633651fdd0..30accc846d2b6 100644 --- a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll +++ b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-any.ll @@ -14,7 +14,7 @@ define amdgpu_kernel void @kern() #0 { ; OBJ-NEXT: 0000 00000000 00000000 00000000 00000000 ................ ; OBJ-NEXT: 0010 00000000 00000000 00000000 00000000 ................ ; OBJ-NEXT: 0020 00000000 00000000 00000000 00000000 ................ -; OBJ-NEXT: 0030 4000af00 88000000 01000000 00000000 @............... +; OBJ-NEXT: 0030 4000af00 8c000000 21000000 00000000 @.......!....... ; ELF: AMDGPU Metadata ; ELF: .sgpr_count: 9 diff --git a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll index 2097579e0c995..4f84b31f1877b 100644 --- a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll +++ b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-off.ll @@ -14,7 +14,7 @@ define amdgpu_kernel void @kern() #0 { ; OBJ-NEXT: 0000 00000000 00000000 00000000 00000000 ................ ; OBJ-NEXT: 0010 00000000 00000000 00000000 00000000 ................ ; OBJ-NEXT: 0020 00000000 00000000 00000000 00000000 ................ -; OBJ-NEXT: 0030 0000af00 88000000 01000000 00000000 ................ +; OBJ-NEXT: 0030 0000af00 8c000000 21000000 00000000 ........!....... ; ELF: AMDGPU Metadata ; ELF: .sgpr_count: 5 diff --git a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll index 775c62e73261a..644f434923368 100644 --- a/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll +++ b/llvm/test/CodeGen/AMDGPU/tid-kd-xnack-on.ll @@ -14,7 +14,7 @@ define amdgpu_kernel void @kern() #0 { ; OBJ-NEXT: 0000 00000000 00000000 00000000 00000000 ................ ; OBJ-NEXT: 0010 00000000 00000000 00000000 00000000 ................ ; OBJ-NEXT: 0020 00000000 00000000 00000000 00000000 ................ -; OBJ-NEXT: 0030 4000af00 88000000 01000000 00000000 @............... +; OBJ-NEXT: 0030 4000af00 8c000000 21000000 00000000 @.......!....... ; ELF: AMDGPU Metadata ; ELF: .sgpr_count: 9 diff --git a/llvm/test/CodeGen/AMDGPU/trap-abis.ll b/llvm/test/CodeGen/AMDGPU/trap-abis.ll index b8f0d7617167e..69cc63eba6243 100644 --- a/llvm/test/CodeGen/AMDGPU/trap-abis.ll +++ b/llvm/test/CodeGen/AMDGPU/trap-abis.ll @@ -23,11 +23,14 @@ define amdgpu_kernel void @trap(ptr addrspace(1) nocapture readonly %arg0) { ; HSA-TRAP-GFX803-LABEL: trap: ; HSA-TRAP-GFX803: ; %bb.0: ; HSA-TRAP-GFX803-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x0 +; HSA-TRAP-GFX803-NEXT: s_add_i32 s12, s12, s17 +; HSA-TRAP-GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; HSA-TRAP-GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v2, 1 -; HSA-TRAP-GFX803-NEXT: s_mov_b64 s[0:1], s[6:7] ; HSA-TRAP-GFX803-NEXT: s_waitcnt lgkmcnt(0) ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v0, s2 ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v1, s3 +; HSA-TRAP-GFX803-NEXT: s_mov_b64 s[0:1], s[6:7] ; HSA-TRAP-GFX803-NEXT: flat_store_dword v[0:1], v2 ; HSA-TRAP-GFX803-NEXT: s_waitcnt vmcnt(0) ; HSA-TRAP-GFX803-NEXT: s_trap 2 @@ -121,6 +124,9 @@ define amdgpu_kernel void @non_entry_trap(ptr addrspace(1) nocapture readonly %a ; HSA-TRAP-GFX803-LABEL: non_entry_trap: ; HSA-TRAP-GFX803: ; %bb.0: ; %entry ; HSA-TRAP-GFX803-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; HSA-TRAP-GFX803-NEXT: s_add_i32 s12, s12, s17 +; HSA-TRAP-GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; HSA-TRAP-GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; HSA-TRAP-GFX803-NEXT: s_waitcnt lgkmcnt(0) ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v0, s0 ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v1, s1 @@ -280,6 +286,9 @@ define amdgpu_kernel void @trap_with_use_after(ptr addrspace(1) %arg0, ptr addrs ; HSA-TRAP-GFX803: ; %bb.0: ; HSA-TRAP-GFX803-NEXT: s_mov_b64 s[0:1], s[6:7] ; HSA-TRAP-GFX803-NEXT: s_load_dwordx4 s[4:7], s[8:9], 0x0 +; HSA-TRAP-GFX803-NEXT: s_add_i32 s12, s12, s17 +; HSA-TRAP-GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; HSA-TRAP-GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; HSA-TRAP-GFX803-NEXT: s_waitcnt lgkmcnt(0) ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v0, s4 ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v1, s5 @@ -411,10 +420,13 @@ define amdgpu_kernel void @debugtrap(ptr addrspace(1) nocapture readonly %arg0) ; HSA-TRAP-GFX803-LABEL: debugtrap: ; HSA-TRAP-GFX803: ; %bb.0: ; HSA-TRAP-GFX803-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; HSA-TRAP-GFX803-NEXT: s_add_i32 s12, s12, s17 +; HSA-TRAP-GFX803-NEXT: s_mov_b32 flat_scratch_lo, s13 +; HSA-TRAP-GFX803-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v2, 1 -; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v3, 2 ; HSA-TRAP-GFX803-NEXT: s_waitcnt lgkmcnt(0) ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v0, s0 +; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v3, 2 ; HSA-TRAP-GFX803-NEXT: v_mov_b32_e32 v1, s1 ; HSA-TRAP-GFX803-NEXT: flat_store_dword v[0:1], v2 ; HSA-TRAP-GFX803-NEXT: s_waitcnt vmcnt(0) diff --git a/llvm/test/CodeGen/AMDGPU/udiv.ll b/llvm/test/CodeGen/AMDGPU/udiv.ll index 6e29536feb51b..660ff4677547a 100644 --- a/llvm/test/CodeGen/AMDGPU/udiv.ll +++ b/llvm/test/CodeGen/AMDGPU/udiv.ll @@ -81,6 +81,9 @@ define amdgpu_kernel void @udiv_i32(ptr addrspace(1) %out, ptr addrspace(1) %in) ; GCN-LABEL: udiv_i32: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s2 ; GCN-NEXT: v_mov_b32_e32 v1, s3 @@ -252,6 +255,9 @@ define amdgpu_kernel void @s_udiv_i32(ptr addrspace(1) %out, i32 %a, i32 %b) { ; GCN-LABEL: s_udiv_i32: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_cvt_f32_u32_e32 v0, s3 ; GCN-NEXT: s_sub_i32 s4, 0, s3 @@ -457,6 +463,9 @@ define amdgpu_kernel void @udiv_v2i32(ptr addrspace(1) %out, ptr addrspace(1) %i ; GCN-LABEL: udiv_v2i32: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s2 ; GCN-NEXT: v_mov_b32_e32 v1, s3 @@ -810,6 +819,9 @@ define amdgpu_kernel void @udiv_v4i32(ptr addrspace(1) %out, ptr addrspace(1) %i ; GCN-LABEL: udiv_v4i32: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_add_u32 s4, s2, 16 ; GCN-NEXT: s_addc_u32 s5, s3, 0 @@ -1135,6 +1147,9 @@ define amdgpu_kernel void @udiv_i32_div_pow2(ptr addrspace(1) %out, ptr addrspac ; GCN-LABEL: udiv_i32_div_pow2: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s2 ; GCN-NEXT: v_mov_b32_e32 v1, s3 @@ -1224,6 +1239,9 @@ define amdgpu_kernel void @udiv_i32_div_k_even(ptr addrspace(1) %out, ptr addrsp ; GCN-LABEL: udiv_i32_div_k_even: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s2 ; GCN-NEXT: v_mov_b32_e32 v1, s3 @@ -1318,6 +1336,9 @@ define amdgpu_kernel void @udiv_i32_div_k_odd(ptr addrspace(1) %out, ptr addrspa ; GCN-LABEL: udiv_i32_div_k_odd: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s2 ; GCN-NEXT: v_mov_b32_e32 v1, s3 @@ -1430,6 +1451,9 @@ define amdgpu_kernel void @v_udiv_i8(ptr addrspace(1) %out, ptr addrspace(1) %in ; GCN-LABEL: v_udiv_i8: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s2 ; GCN-NEXT: v_mov_b32_e32 v1, s3 @@ -1570,6 +1594,9 @@ define amdgpu_kernel void @v_udiv_i16(ptr addrspace(1) %out, ptr addrspace(1) %i ; GCN-LABEL: v_udiv_i16: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s2 ; GCN-NEXT: v_mov_b32_e32 v1, s3 @@ -1726,6 +1753,9 @@ define amdgpu_kernel void @v_udiv_i23(ptr addrspace(1) %out, ptr addrspace(1) %i ; GCN-LABEL: v_udiv_i23: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_add_u32 s4, s2, 4 ; GCN-NEXT: s_addc_u32 s5, s3, 0 @@ -1923,6 +1953,9 @@ define amdgpu_kernel void @v_udiv_i24(ptr addrspace(1) %out, ptr addrspace(1) %i ; GCN-LABEL: v_udiv_i24: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_add_u32 s4, s2, 4 ; GCN-NEXT: s_addc_u32 s5, s3, 0 @@ -2105,6 +2138,9 @@ define amdgpu_kernel void @scalarize_mulhu_4xi32(ptr addrspace(1) nocapture read ; GCN-LABEL: scalarize_mulhu_4xi32: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s0 ; GCN-NEXT: v_mov_b32_e32 v1, s1 @@ -2218,6 +2254,9 @@ define amdgpu_kernel void @test_udiv2(i32 %p) { ; GCN-LABEL: test_udiv2: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dword s0, s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: s_lshr_b32 s0, s0, 1 ; GCN-NEXT: v_mov_b32_e32 v0, s0 @@ -2281,6 +2320,9 @@ define amdgpu_kernel void @test_udiv_3_mulhu(i32 %p) { ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dword s0, s[8:9], 0x0 ; GCN-NEXT: v_mov_b32_e32 v0, 0xaaaaaaab +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mul_hi_u32 v0, s0, v0 ; GCN-NEXT: v_lshrrev_b32_e32 v0, 1, v0 @@ -2378,6 +2420,9 @@ define amdgpu_kernel void @fdiv_test_denormals(ptr addrspace(1) nocapture readon ; GCN-LABEL: fdiv_test_denormals: ; GCN: ; %bb.0: ; %bb ; GCN-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_mov_b32_e32 v0, s0 ; GCN-NEXT: v_mov_b32_e32 v1, s1 diff --git a/llvm/test/CodeGen/AMDGPU/uint_to_fp.f64.ll b/llvm/test/CodeGen/AMDGPU/uint_to_fp.f64.ll index 55cbc14a46706..97738a7944741 100644 --- a/llvm/test/CodeGen/AMDGPU/uint_to_fp.f64.ll +++ b/llvm/test/CodeGen/AMDGPU/uint_to_fp.f64.ll @@ -9,6 +9,9 @@ define amdgpu_kernel void @v_uint_to_fp_i64_to_f64(ptr addrspace(1) %out, ptr ad ; SI: ; %bb.0: ; SI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; SI-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: v_mov_b32_e32 v1, s3 ; SI-NEXT: v_add_i32_e32 v0, vcc, s2, v0 @@ -28,6 +31,9 @@ define amdgpu_kernel void @v_uint_to_fp_i64_to_f64(ptr addrspace(1) %out, ptr ad ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 ; VI-NEXT: v_lshlrev_b32_e32 v0, 3, v0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_mov_b32_e32 v1, s3 ; VI-NEXT: v_add_u32_e32 v0, vcc, s2, v0 @@ -54,6 +60,9 @@ define amdgpu_kernel void @s_uint_to_fp_i64_to_f64(ptr addrspace(1) %out, i64 %i ; SI-LABEL: s_uint_to_fp_i64_to_f64: ; SI: ; %bb.0: ; SI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: v_cvt_f64_u32_e32 v[0:1], s3 ; SI-NEXT: v_cvt_f64_u32_e32 v[2:3], s2 @@ -67,6 +76,9 @@ define amdgpu_kernel void @s_uint_to_fp_i64_to_f64(ptr addrspace(1) %out, i64 %i ; VI-LABEL: s_uint_to_fp_i64_to_f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f64_u32_e32 v[0:1], s3 ; VI-NEXT: v_cvt_f64_u32_e32 v[2:3], s2 @@ -86,6 +98,9 @@ define amdgpu_kernel void @s_uint_to_fp_v2i64_to_v2f64(ptr addrspace(1) %out, <2 ; SI: ; %bb.0: ; SI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x4 ; SI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: v_cvt_f64_u32_e32 v[0:1], s3 ; SI-NEXT: v_cvt_f64_u32_e32 v[2:3], s1 @@ -103,6 +118,9 @@ define amdgpu_kernel void @s_uint_to_fp_v2i64_to_v2f64(ptr addrspace(1) %out, <2 ; VI-LABEL: s_uint_to_fp_v2i64_to_v2f64: ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x10 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f64_u32_e32 v[0:1], s3 ; VI-NEXT: v_cvt_f64_u32_e32 v[2:3], s1 @@ -128,6 +146,9 @@ define amdgpu_kernel void @s_uint_to_fp_v4i64_to_v4f64(ptr addrspace(1) %out, <4 ; SI: ; %bb.0: ; SI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x8 ; SI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: v_cvt_f64_u32_e32 v[0:1], s3 ; SI-NEXT: v_cvt_f64_u32_e32 v[4:5], s1 @@ -160,6 +181,9 @@ define amdgpu_kernel void @s_uint_to_fp_v4i64_to_v4f64(ptr addrspace(1) %out, <4 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx8 s[0:7], s[8:9], 0x20 ; VI-NEXT: s_load_dwordx2 s[8:9], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f64_u32_e32 v[2:3], s7 ; VI-NEXT: v_cvt_f64_u32_e32 v[4:5], s5 @@ -196,6 +220,9 @@ define amdgpu_kernel void @s_uint_to_fp_i32_to_f64(ptr addrspace(1) %out, i32 %i ; SI: ; %bb.0: ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: v_cvt_f64_u32_e32 v[0:1], s2 ; SI-NEXT: v_mov_b32_e32 v3, s1 @@ -207,6 +234,9 @@ define amdgpu_kernel void @s_uint_to_fp_i32_to_f64(ptr addrspace(1) %out, i32 %i ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f64_u32_e32 v[0:1], s2 ; VI-NEXT: v_mov_b32_e32 v3, s1 @@ -222,6 +252,9 @@ define amdgpu_kernel void @s_uint_to_fp_v2i32_to_v2f64(ptr addrspace(1) %out, <2 ; GCN-LABEL: s_uint_to_fp_v2i32_to_v2f64: ; GCN: ; %bb.0: ; GCN-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0 +; GCN-NEXT: s_add_i32 s12, s12, s17 +; GCN-NEXT: s_mov_b32 flat_scratch_lo, s13 +; GCN-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; GCN-NEXT: s_waitcnt lgkmcnt(0) ; GCN-NEXT: v_cvt_f64_u32_e32 v[2:3], s3 ; GCN-NEXT: v_cvt_f64_u32_e32 v[0:1], s2 @@ -239,6 +272,9 @@ define amdgpu_kernel void @s_uint_to_fp_v4i32_to_v4f64(ptr addrspace(1) %out, <4 ; SI: ; %bb.0: ; SI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x4 ; SI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: v_cvt_f64_u32_e32 v[0:1], s0 ; SI-NEXT: v_cvt_f64_u32_e32 v[6:7], s3 @@ -259,6 +295,9 @@ define amdgpu_kernel void @s_uint_to_fp_v4i32_to_v4f64(ptr addrspace(1) %out, <4 ; VI: ; %bb.0: ; VI-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x10 ; VI-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: v_cvt_f64_u32_e32 v[0:1], s0 ; VI-NEXT: v_cvt_f64_u32_e32 v[6:7], s3 @@ -286,11 +325,14 @@ define amdgpu_kernel void @uint_to_fp_i1_to_f64(ptr addrspace(1) %out, i32 %in) ; SI: ; %bb.0: ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; SI-NEXT: v_mov_b32_e32 v0, 0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_cmp_eq_u32 s2, 0 ; SI-NEXT: s_cselect_b32 s2, 0x3ff00000, 0 ; SI-NEXT: v_mov_b32_e32 v3, s1 +; SI-NEXT: v_mov_b32_e32 v0, 0 ; SI-NEXT: v_mov_b32_e32 v1, s2 ; SI-NEXT: v_mov_b32_e32 v2, s0 ; SI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -300,11 +342,14 @@ define amdgpu_kernel void @uint_to_fp_i1_to_f64(ptr addrspace(1) %out, i32 %in) ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0x3ff00000, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -320,6 +365,9 @@ define amdgpu_kernel void @uint_to_fp_i1_to_f64_load(ptr addrspace(1) %out, i1 % ; SI: ; %bb.0: ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_bitcmp1_b32 s2, 0 ; SI-NEXT: s_cselect_b64 s[2:3], -1, 0 @@ -334,6 +382,9 @@ define amdgpu_kernel void @uint_to_fp_i1_to_f64_load(ptr addrspace(1) %out, i1 % ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_bitcmp1_b32 s2, 0 ; VI-NEXT: s_cselect_b64 s[2:3], -1, 0 @@ -353,6 +404,9 @@ define amdgpu_kernel void @s_uint_to_fp_i8_to_f64(ptr addrspace(1) %out, i8 %in) ; SI: ; %bb.0: ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_and_b32 s2, s2, 0xff ; SI-NEXT: v_cvt_f64_u32_e32 v[0:1], s2 @@ -365,6 +419,9 @@ define amdgpu_kernel void @s_uint_to_fp_i8_to_f64(ptr addrspace(1) %out, i8 %in) ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_and_b32 s2, s2, 0xff ; VI-NEXT: v_cvt_f64_u32_e32 v[0:1], s2 @@ -402,11 +459,14 @@ define amdgpu_kernel void @s_select_uint_to_fp_i1_vals_f64(ptr addrspace(1) %out ; SI: ; %bb.0: ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; SI-NEXT: v_mov_b32_e32 v0, 0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_cmp_eq_u32 s2, 0 ; SI-NEXT: s_cselect_b32 s2, 0x3ff00000, 0 ; SI-NEXT: v_mov_b32_e32 v3, s1 +; SI-NEXT: v_mov_b32_e32 v0, 0 ; SI-NEXT: v_mov_b32_e32 v1, s2 ; SI-NEXT: v_mov_b32_e32 v2, s0 ; SI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -416,11 +476,14 @@ define amdgpu_kernel void @s_select_uint_to_fp_i1_vals_f64(ptr addrspace(1) %out ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0x3ff00000, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -453,11 +516,14 @@ define amdgpu_kernel void @s_select_uint_to_fp_i1_vals_i64(ptr addrspace(1) %out ; SI: ; %bb.0: ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; SI-NEXT: v_mov_b32_e32 v0, 0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_cmp_eq_u32 s2, 0 ; SI-NEXT: s_cselect_b32 s2, 0x3ff00000, 0 ; SI-NEXT: v_mov_b32_e32 v3, s1 +; SI-NEXT: v_mov_b32_e32 v0, 0 ; SI-NEXT: v_mov_b32_e32 v1, s2 ; SI-NEXT: v_mov_b32_e32 v2, s0 ; SI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -467,11 +533,14 @@ define amdgpu_kernel void @s_select_uint_to_fp_i1_vals_i64(ptr addrspace(1) %out ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0x3ff00000, 0 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -505,11 +574,14 @@ define amdgpu_kernel void @s_swap_select_uint_to_fp_i1_vals_f64(ptr addrspace(1) ; SI: ; %bb.0: ; SI-NEXT: s_load_dword s2, s[8:9], 0x2 ; SI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; SI-NEXT: v_mov_b32_e32 v0, 0 +; SI-NEXT: s_add_i32 s12, s12, s17 +; SI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; SI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; SI-NEXT: s_waitcnt lgkmcnt(0) ; SI-NEXT: s_cmp_eq_u32 s2, 0 ; SI-NEXT: s_cselect_b32 s2, 0, 0x3ff00000 ; SI-NEXT: v_mov_b32_e32 v3, s1 +; SI-NEXT: v_mov_b32_e32 v0, 0 ; SI-NEXT: v_mov_b32_e32 v1, s2 ; SI-NEXT: v_mov_b32_e32 v2, s0 ; SI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] @@ -519,11 +591,14 @@ define amdgpu_kernel void @s_swap_select_uint_to_fp_i1_vals_f64(ptr addrspace(1) ; VI: ; %bb.0: ; VI-NEXT: s_load_dword s2, s[8:9], 0x8 ; VI-NEXT: s_load_dwordx2 s[0:1], s[8:9], 0x0 -; VI-NEXT: v_mov_b32_e32 v0, 0 +; VI-NEXT: s_add_i32 s12, s12, s17 +; VI-NEXT: s_lshr_b32 flat_scratch_hi, s12, 8 +; VI-NEXT: s_mov_b32 flat_scratch_lo, s13 ; VI-NEXT: s_waitcnt lgkmcnt(0) ; VI-NEXT: s_cmp_eq_u32 s2, 0 ; VI-NEXT: s_cselect_b32 s2, 0, 0x3ff00000 ; VI-NEXT: v_mov_b32_e32 v3, s1 +; VI-NEXT: v_mov_b32_e32 v0, 0 ; VI-NEXT: v_mov_b32_e32 v1, s2 ; VI-NEXT: v_mov_b32_e32 v2, s0 ; VI-NEXT: flat_store_dwordx2 v[2:3], v[0:1] diff --git a/llvm/test/CodeGen/AMDGPU/vgpr-spill-placement-issue61083.ll b/llvm/test/CodeGen/AMDGPU/vgpr-spill-placement-issue61083.ll index 45ea6b62761cc..ab7e85fdff516 100644 --- a/llvm/test/CodeGen/AMDGPU/vgpr-spill-placement-issue61083.ll +++ b/llvm/test/CodeGen/AMDGPU/vgpr-spill-placement-issue61083.ll @@ -11,7 +11,7 @@ define amdgpu_kernel void @__omp_offloading_16_dd2df_main_l9() { ; CHECK-LABEL: __omp_offloading_16_dd2df_main_l9: ; CHECK: ; %bb.0: ; %bb -; CHECK-NEXT: s_add_u32 s0, s0, s15 +; CHECK-NEXT: s_add_u32 s0, s0, s17 ; CHECK-NEXT: s_addc_u32 s1, s1, 0 ; CHECK-NEXT: v_mov_b32_e32 v1, v0 ; CHECK-NEXT: v_mov_b32_e32 v0, 0 diff --git a/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg-debug.ll b/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg-debug.ll index e99a06f497016..1bc25a1386074 100644 --- a/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg-debug.ll +++ b/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg-debug.ll @@ -95,7 +95,7 @@ ; Function Attrs: convergent nocallback nofree nounwind willreturn declare void @llvm.amdgcn.end.cf.i64(i64) #2 - attributes #0 = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } + attributes #0 = { "amdgpu-no-completion-action" "amdgpu-no-default-queue" "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-heap-ptr" "amdgpu-no-hostcall-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-multigrid-sync-arg" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" "uniform-work-group-size"="false" } attributes #1 = { nocallback nofree nosync nounwind speculatable willreturn memory(none) } attributes #2 = { convergent nocallback nofree nounwind willreturn } attributes #3 = { convergent nocallback nofree nounwind willreturn memory(none) } diff --git a/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg.ll b/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg.ll index 076d7c9cd8842..0515ffa094329 100644 --- a/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg.ll +++ b/llvm/test/CodeGen/MIR/AMDGPU/machine-function-info-long-branch-reg.ll @@ -69,5 +69,5 @@ bb4: ret void } -attributes #0 = { nounwind "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" } +attributes #0 = { nounwind "amdgpu-no-dispatch-id" "amdgpu-no-dispatch-ptr" "amdgpu-no-flat-scratch-init" "amdgpu-no-implicitarg-ptr" "amdgpu-no-lds-kernel-id" "amdgpu-no-queue-ptr" "amdgpu-no-workgroup-id-x" "amdgpu-no-workgroup-id-y" "amdgpu-no-workgroup-id-z" "amdgpu-no-workitem-id-x" "amdgpu-no-workitem-id-y" "amdgpu-no-workitem-id-z" } attributes #1 = { nounwind readnone } From 16980d5463c787a48ffb78fd9bbe3d9d32757f34 Mon Sep 17 00:00:00 2001 From: Matheus Izvekov Date: Tue, 15 Apr 2025 19:34:55 -0300 Subject: [PATCH 39/63] Revert "[Clang] Fix dependent local class instantiation bugs" (#135870) Reverts llvm/llvm-project#134038 This crashes clang as reported here: https://github.com/llvm/llvm-project/pull/134038#issuecomment-2807092646 --- clang/docs/ReleaseNotes.rst | 1 - clang/lib/Sema/SemaTemplateInstantiate.cpp | 3 + .../lib/Sema/SemaTemplateInstantiateDecl.cpp | 56 +--------------- .../CodeGenCXX/local-class-instantiation.cpp | 64 ------------------- 4 files changed, 4 insertions(+), 120 deletions(-) delete mode 100644 clang/test/CodeGenCXX/local-class-instantiation.cpp diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst index 166f26921cb71..ee69af5632f6e 100644 --- a/clang/docs/ReleaseNotes.rst +++ b/clang/docs/ReleaseNotes.rst @@ -457,7 +457,6 @@ Bug Fixes to C++ Support by template argument deduction. - Clang is now better at instantiating the function definition after its use inside of a constexpr lambda. (#GH125747) -- Fixed a local class member function instantiation bug inside dependent lambdas. (#GH59734), (#GH132208) - Clang no longer crashes when trying to unify the types of arrays with certain differences in qualifiers (this could happen during template argument deduction or when building a ternary operator). (#GH97005) diff --git a/clang/lib/Sema/SemaTemplateInstantiate.cpp b/clang/lib/Sema/SemaTemplateInstantiate.cpp index 0e81804f8c1e7..d2408a94ad0ab 100644 --- a/clang/lib/Sema/SemaTemplateInstantiate.cpp +++ b/clang/lib/Sema/SemaTemplateInstantiate.cpp @@ -4126,6 +4126,9 @@ Sema::InstantiateClassMembers(SourceLocation PointOfInstantiation, if (FunctionDecl *Pattern = Function->getInstantiatedFromMemberFunction()) { + if (Function->isIneligibleOrNotSelected()) + continue; + if (Function->getTrailingRequiresClause()) { ConstraintSatisfaction Satisfaction; if (CheckFunctionConstraints(Function, Satisfaction) || diff --git a/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp b/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp index bf5a882ba4f12..5c80077f294c6 100644 --- a/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp +++ b/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp @@ -5597,61 +5597,7 @@ void Sema::InstantiateFunctionDefinition(SourceLocation PointOfInstantiation, Function->setLocation(PatternDecl->getLocation()); Function->setInnerLocStart(PatternDecl->getInnerLocStart()); Function->setRangeEnd(PatternDecl->getEndLoc()); - // Let the instantiation use the Pattern's DeclarationNameLoc, due to the - // following awkwardness: - // - // 1. There are out-of-tree users of getNameInfo().getSourceRange(), who - // expect the source range of the instantiated declaration to be set to - // point to the definition. - // - // 2. That getNameInfo().getSourceRange() might return the TypeLocInfo's - // location it tracked. - // - // 3. Function might come from an (implicit) declaration, while the pattern - // comes from a definition. In these cases, we need the PatternDecl's source - // location. - // - // To that end, we need to more or less tweak the DeclarationNameLoc. However, - // we can't blindly copy the DeclarationNameLoc from the PatternDecl to the - // function, since it contains associated TypeLocs that should have already - // been transformed. So, we rebuild the TypeLoc for that purpose. Technically, - // we should create a new function declaration and assign everything we need, - // but InstantiateFunctionDefinition updates the declaration in place. - auto NameLocPointsToPattern = [&] { - DeclarationNameInfo PatternName = PatternDecl->getNameInfo(); - DeclarationNameLoc PatternNameLoc = PatternName.getInfo(); - switch (PatternName.getName().getNameKind()) { - case DeclarationName::CXXConstructorName: - case DeclarationName::CXXDestructorName: - case DeclarationName::CXXConversionFunctionName: - break; - default: - // Cases where DeclarationNameLoc doesn't matter, as it merely contains a - // source range. - return PatternNameLoc; - } - - TypeSourceInfo *TSI = Function->getNameInfo().getNamedTypeInfo(); - // TSI might be null if the function is named by a constructor template id. - // E.g. S() {} for class template S with a template parameter T. - if (!TSI) { - // We don't care about the DeclarationName of the instantiated function, - // but only the DeclarationNameLoc. So if the TypeLoc is absent, we do - // nothing. - return PatternNameLoc; - } - - QualType InstT = TSI->getType(); - // We want to use a TypeLoc that reflects the transformed type while - // preserving the source location from the pattern. - TypeLocBuilder TLB; - TLB.pushTrivial( - Context, InstT, - PatternNameLoc.getNamedTypeInfo()->getTypeLoc().getBeginLoc()); - return DeclarationNameLoc::makeNamedTypeLoc( - TLB.getTypeSourceInfo(Context, InstT)); - }; - Function->setDeclarationNameLoc(NameLocPointsToPattern()); + Function->setDeclarationNameLoc(PatternDecl->getNameInfo().getInfo()); EnterExpressionEvaluationContext EvalContext( *this, Sema::ExpressionEvaluationContext::PotentiallyEvaluated); diff --git a/clang/test/CodeGenCXX/local-class-instantiation.cpp b/clang/test/CodeGenCXX/local-class-instantiation.cpp deleted file mode 100644 index 34103a1ee55ef..0000000000000 --- a/clang/test/CodeGenCXX/local-class-instantiation.cpp +++ /dev/null @@ -1,64 +0,0 @@ -// RUN: %clang_cc1 -std=c++17 %s -emit-llvm -triple %itanium_abi_triple -o - | FileCheck %s - -namespace LambdaContainingLocalClasses { - -template -void GH59734() { - [&](auto param) { - struct Guard { - Guard() { - // Check that we're able to create DeclRefExpr to param at this point. - static_assert(__is_same(decltype(param), int), ""); - } - ~Guard() { - static_assert(__is_same(decltype(param), int), ""); - } - operator decltype(param)() { - return decltype(param)(); - } - }; - Guard guard; - param = guard; - }(42); -} - -// Guard::Guard(): -// CHECK-DAG: define {{.*}} @_ZZZN28LambdaContainingLocalClasses7GH59734IiEEvvENKUlT_E_clIiEEDaS1_EN5GuardC2Ev -// Guard::operator int(): -// CHECK-DAG: define {{.*}} @_ZZZN28LambdaContainingLocalClasses7GH59734IiEEvvENKUlT_E_clIiEEDaS1_EN5GuardcviEv -// Guard::~Guard(): -// CHECK-DAG: define {{.*}} @_ZZZN28LambdaContainingLocalClasses7GH59734IiEEvvENKUlT_E_clIiEEDaS1_EN5GuardD2Ev - -struct S {}; - -template -auto GH132208 = [](auto param) { - struct OnScopeExit { - OnScopeExit() { - static_assert(__is_same(decltype(param), S), ""); - } - ~OnScopeExit() { - static_assert(__is_same(decltype(param), S), ""); - } - operator decltype(param)() { - return decltype(param)(); - } - } pending; - - param = pending; -}; - -void bar() { - GH59734(); - - GH132208(S{}); -} - -// OnScopeExit::OnScopeExit(): -// CHECK-DAG: define {{.*}} @_ZZNK28LambdaContainingLocalClasses8GH132208IvEMUlT_E_clINS_1SEEEDaS2_EN11OnScopeExitC2Ev -// OnScopeExit::operator S(): -// CHECK-DAG: define {{.*}} @_ZZNK28LambdaContainingLocalClasses8GH132208IvEMUlT_E_clINS_1SEEEDaS2_EN11OnScopeExitcvS5_Ev -// OnScopeExit::~OnScopeExit(): -// CHECK-DAG: define {{.*}} @_ZZNK28LambdaContainingLocalClasses8GH132208IvEMUlT_E_clINS_1SEEEDaS2_EN11OnScopeExitD2Ev - -} // namespace LambdaContainingLocalClasses From af63e1b505453de3e6a281d1b72e62fa8d396b23 Mon Sep 17 00:00:00 2001 From: Erich Keane Date: Tue, 15 Apr 2025 15:52:04 -0700 Subject: [PATCH 40/63] [OpenACC][CIR] Implement 'self' lowering on compute constructs (#135851) This is our first attempt at lowering a clause that is an 'operand' in the OpenACC operand, so it does quite a bit of refactoring. My previous plans on how to emit the clauses was not viable, so we instead do 'create the op, then use the visitor to fill in the operands'. This resulted in the 'applyAttributes' function getting removed and a few other functions simplified. Additionally, it requires setting the insertion point a little to make sure we're inserting 'around' the operation correctly. Finally, since the OpenACC dialect only understands the MLIR types, we had to introduce a use of the unrealized-conversion-cast, which we'll probably getting good use out of in the future. --- clang/include/clang/AST/OpenACCClause.h | 5 + clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp | 216 +++++++++++--------- clang/test/CIR/CodeGenOpenACC/kernels.c | 30 ++- clang/test/CIR/CodeGenOpenACC/parallel.c | 30 ++- clang/test/CIR/CodeGenOpenACC/serial.c | 30 ++- 5 files changed, 205 insertions(+), 106 deletions(-) diff --git a/clang/include/clang/AST/OpenACCClause.h b/clang/include/clang/AST/OpenACCClause.h index 3687af76a559f..681567228cbb0 100644 --- a/clang/include/clang/AST/OpenACCClause.h +++ b/clang/include/clang/AST/OpenACCClause.h @@ -430,6 +430,11 @@ class OpenACCSelfClause final } bool isConditionExprClause() const { return HasConditionExpr.has_value(); } + bool isVarListClause() const { return !isConditionExprClause(); } + bool isEmptySelfClause() const { + return (isConditionExprClause() && !hasConditionExpr()) || + (!isConditionExprClause() && getVarList().empty()); + } bool hasConditionExpr() const { assert(HasConditionExpr.has_value() && diff --git a/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp b/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp index 152f996ed0fed..3bcc6f908a841 100644 --- a/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp +++ b/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp @@ -32,46 +32,51 @@ constexpr bool isOneOfTypes = template constexpr bool isOneOfTypes = std::is_same_v; +template class OpenACCClauseCIREmitter final - : public OpenACCClauseVisitor { - CIRGenModule &cgm; + : public OpenACCClauseVisitor> { + OpTy &operation; + CIRGenFunction &cgf; + CIRGenBuilderTy &builder; + // This is necessary since a few of the clauses emit differently based on the // directive kind they are attached to. OpenACCDirectiveKind dirKind; + // TODO(cir): This source location should be able to go away once the NYI + // diagnostics are gone. SourceLocation dirLoc; - struct AttributeData { - // Value of the 'default' attribute, added on 'data' and 'compute'/etc - // constructs as a 'default-attr'. - std::optional defaultVal = std::nullopt; - // For directives that have their device type architectures listed in - // attributes (init/shutdown/etc), the list of architectures to be emitted. - llvm::SmallVector deviceTypeArchs{}; - } attrData; - void clauseNotImplemented(const OpenACCClause &c) { - cgm.errorNYI(c.getSourceRange(), "OpenACC Clause", c.getClauseKind()); + cgf.cgm.errorNYI(c.getSourceRange(), "OpenACC Clause", c.getClauseKind()); } public: - OpenACCClauseCIREmitter(CIRGenModule &cgm, OpenACCDirectiveKind dirKind, - SourceLocation dirLoc) - : cgm(cgm), dirKind(dirKind), dirLoc(dirLoc) {} + OpenACCClauseCIREmitter(OpTy &operation, CIRGenFunction &cgf, + CIRGenBuilderTy &builder, + OpenACCDirectiveKind dirKind, SourceLocation dirLoc) + : operation(operation), cgf(cgf), builder(builder), dirKind(dirKind), + dirLoc(dirLoc) {} void VisitClause(const OpenACCClause &clause) { clauseNotImplemented(clause); } void VisitDefaultClause(const OpenACCDefaultClause &clause) { - switch (clause.getDefaultClauseKind()) { - case OpenACCDefaultClauseKind::None: - attrData.defaultVal = ClauseDefaultValue::None; - break; - case OpenACCDefaultClauseKind::Present: - attrData.defaultVal = ClauseDefaultValue::Present; - break; - case OpenACCDefaultClauseKind::Invalid: - break; + // This type-trait checks if 'op'(the first arg) is one of the mlir::acc + // operations listed in the rest of the arguments. + if constexpr (isOneOfTypes) { + switch (clause.getDefaultClauseKind()) { + case OpenACCDefaultClauseKind::None: + operation.setDefaultAttr(ClauseDefaultValue::None); + break; + case OpenACCDefaultClauseKind::Present: + operation.setDefaultAttr(ClauseDefaultValue::Present); + break; + case OpenACCDefaultClauseKind::Invalid: + break; + } + } else { + return clauseNotImplemented(clause); } } @@ -89,64 +94,70 @@ class OpenACCClauseCIREmitter final } void VisitDeviceTypeClause(const OpenACCDeviceTypeClause &clause) { + if constexpr (isOneOfTypes) { + llvm::SmallVector deviceTypes; + std::optional existingDeviceTypes = + operation.getDeviceTypes(); + + // Ensure we keep the existing ones, and in the correct 'new' order. + if (existingDeviceTypes) { + for (const mlir::Attribute &Attr : *existingDeviceTypes) + deviceTypes.push_back(mlir::acc::DeviceTypeAttr::get( + builder.getContext(), + cast(Attr).getValue())); + } - switch (dirKind) { - case OpenACCDirectiveKind::Init: - case OpenACCDirectiveKind::Set: - case OpenACCDirectiveKind::Shutdown: { - // Device type has a list that is either a 'star' (emitted as 'star'), - // or an identifer list, all of which get added for attributes. - - for (const DeviceTypeArgument &arg : clause.getArchitectures()) - attrData.deviceTypeArchs.push_back(decodeDeviceType(arg.first)); - break; - } - default: + for (const DeviceTypeArgument &arg : clause.getArchitectures()) { + deviceTypes.push_back(mlir::acc::DeviceTypeAttr::get( + builder.getContext(), decodeDeviceType(arg.first))); + } + operation.removeDeviceTypesAttr(); + operation.setDeviceTypesAttr( + mlir::ArrayAttr::get(builder.getContext(), deviceTypes)); + } else if constexpr (isOneOfTypes) { + assert(!operation.getDeviceTypeAttr() && "already have device-type?"); + assert(clause.getArchitectures().size() <= 1); + + if (!clause.getArchitectures().empty()) + operation.setDeviceType( + decodeDeviceType(clause.getArchitectures()[0].first)); + } else { return clauseNotImplemented(clause); } } - // Apply any of the clauses that resulted in an 'attribute'. - template - void applyAttributes(CIRGenBuilderTy &builder, Op &op) { - - if (attrData.defaultVal.has_value()) { - // FIXME: OpenACC: as we implement this for other directive kinds, we have - // to expand this list. - // This type-trait checks if 'op'(the first arg) is one of the mlir::acc - // operations listed in the rest of the arguments. - if constexpr (isOneOfTypes) - op.setDefaultAttr(*attrData.defaultVal); - else - cgm.errorNYI(dirLoc, "OpenACC 'default' clause lowering for ", dirKind); - } - - if (!attrData.deviceTypeArchs.empty()) { - // FIXME: OpenACC: as we implement this for other directive kinds, we have - // to expand this list, or more likely, have a 'noop' branch as most other - // uses of this apply to the operands instead. - // This type-trait checks if 'op'(the first arg) is one of the mlir::acc - if constexpr (isOneOfTypes) { - llvm::SmallVector deviceTypes; - for (mlir::acc::DeviceType DT : attrData.deviceTypeArchs) - deviceTypes.push_back( - mlir::acc::DeviceTypeAttr::get(builder.getContext(), DT)); - - op.setDeviceTypesAttr( - mlir::ArrayAttr::get(builder.getContext(), deviceTypes)); - } else if constexpr (isOneOfTypes) { - assert(attrData.deviceTypeArchs.size() <= 1 && - "Set can only have a single architecture"); - if (!attrData.deviceTypeArchs.empty()) - op.setDeviceType(attrData.deviceTypeArchs[0]); + void VisitSelfClause(const OpenACCSelfClause &clause) { + if constexpr (isOneOfTypes) { + if (clause.isEmptySelfClause()) { + operation.setSelfAttr(true); + } else if (clause.isConditionExprClause()) { + assert(clause.hasConditionExpr()); + mlir::Value condition = + cgf.evaluateExprAsBool(clause.getConditionExpr()); + + mlir::Location exprLoc = + cgf.cgm.getLoc(clause.getConditionExpr()->getBeginLoc()); + mlir::IntegerType targetType = mlir::IntegerType::get( + &cgf.getMLIRContext(), /*width=*/1, + mlir::IntegerType::SignednessSemantics::Signless); + auto conversionOp = builder.create( + exprLoc, targetType, condition); + operation.getSelfCondMutable().append(conversionOp.getResult(0)); } else { - cgm.errorNYI(dirLoc, "OpenACC 'device_type' clause lowering for ", - dirKind); + llvm_unreachable("var-list version of self shouldn't get here"); } + } else { + return clauseNotImplemented(clause); } } }; +template +auto makeClauseEmitter(OpTy &op, CIRGenFunction &cgf, CIRGenBuilderTy &builder, + OpenACCDirectiveKind dirKind, SourceLocation dirLoc) { + return OpenACCClauseCIREmitter(op, cgf, builder, dirKind, dirLoc); +} + } // namespace template @@ -158,24 +169,27 @@ mlir::LogicalResult CIRGenFunction::emitOpenACCOpAssociatedStmt( llvm::SmallVector retTy; llvm::SmallVector operands; - - // Clause-emitter must be here because it might modify operands. - OpenACCClauseCIREmitter clauseEmitter(getCIRGenModule(), dirKind, dirLoc); - clauseEmitter.VisitClauseList(clauses); - auto op = builder.create(start, retTy, operands); - // Apply the attributes derived from the clauses. - clauseEmitter.applyAttributes(builder, op); + { + mlir::OpBuilder::InsertionGuard guardCase(builder); + // Sets insertion point before the 'op', since every new expression needs to + // be before the operation. + builder.setInsertionPoint(op); + makeClauseEmitter(op, *this, builder, dirKind, dirLoc) + .VisitClauseList(clauses); + } - mlir::Block &block = op.getRegion().emplaceBlock(); - mlir::OpBuilder::InsertionGuard guardCase(builder); - builder.setInsertionPointToEnd(&block); + { + mlir::Block &block = op.getRegion().emplaceBlock(); + mlir::OpBuilder::InsertionGuard guardCase(builder); + builder.setInsertionPointToEnd(&block); - LexicalScope ls{*this, start, builder.getInsertionBlock()}; - res = emitStmt(associatedStmt, /*useCurrentScope=*/true); + LexicalScope ls{*this, start, builder.getInsertionBlock()}; + res = emitStmt(associatedStmt, /*useCurrentScope=*/true); - builder.create(end); + builder.create(end); + } return res; } @@ -187,14 +201,16 @@ mlir::LogicalResult CIRGenFunction::emitOpenACCOp( llvm::SmallVector retTy; llvm::SmallVector operands; - - // Clause-emitter must be here because it might modify operands. - OpenACCClauseCIREmitter clauseEmitter(getCIRGenModule(), dirKind, dirLoc); - clauseEmitter.VisitClauseList(clauses); - auto op = builder.create(start, retTy, operands); - // Apply the attributes derived from the clauses. - clauseEmitter.applyAttributes(builder, op); + + { + mlir::OpBuilder::InsertionGuard guardCase(builder); + // Sets insertion point before the 'op', since every new expression needs to + // be before the operation. + builder.setInsertionPoint(op); + makeClauseEmitter(op, *this, builder, dirKind, dirLoc) + .VisitClauseList(clauses); + } return res; } @@ -254,46 +270,46 @@ mlir::LogicalResult CIRGenFunction::emitOpenACCShutdownConstruct( mlir::LogicalResult CIRGenFunction::emitOpenACCLoopConstruct(const OpenACCLoopConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC Loop Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC Loop Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCCombinedConstruct( const OpenACCCombinedConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC Combined Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC Combined Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCEnterDataConstruct( const OpenACCEnterDataConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC EnterData Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC EnterData Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCExitDataConstruct( const OpenACCExitDataConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC ExitData Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC ExitData Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCHostDataConstruct( const OpenACCHostDataConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC HostData Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC HostData Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCWaitConstruct(const OpenACCWaitConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC Wait Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC Wait Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCUpdateConstruct(const OpenACCUpdateConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC Update Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC Update Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCAtomicConstruct(const OpenACCAtomicConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC Atomic Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC Atomic Construct"); return mlir::failure(); } mlir::LogicalResult CIRGenFunction::emitOpenACCCacheConstruct(const OpenACCCacheConstruct &s) { - getCIRGenModule().errorNYI(s.getSourceRange(), "OpenACC Cache Construct"); + cgm.errorNYI(s.getSourceRange(), "OpenACC Cache Construct"); return mlir::failure(); } diff --git a/clang/test/CIR/CodeGenOpenACC/kernels.c b/clang/test/CIR/CodeGenOpenACC/kernels.c index 0c950fe3d0f9c..934daf9e8ecc0 100644 --- a/clang/test/CIR/CodeGenOpenACC/kernels.c +++ b/clang/test/CIR/CodeGenOpenACC/kernels.c @@ -1,7 +1,9 @@ // RUN: %clang_cc1 -fopenacc -emit-cir -fclangir %s -o - | FileCheck %s -void acc_kernels(void) { - // CHECK: cir.func @acc_kernels() { +void acc_kernels(int cond) { + // CHECK: cir.func @acc_kernels(%[[ARG:.*]]: !s32i{{.*}}) { + // CHECK-NEXT: %[[COND:.*]] = cir.alloca !s32i, !cir.ptr, ["cond", init] + // CHECK-NEXT: cir.store %[[ARG]], %[[COND]] : !s32i, !cir.ptr #pragma acc kernels {} @@ -38,5 +40,29 @@ void acc_kernels(void) { // CHECK-NEXT: acc.terminator // CHECK-NEXT:} +#pragma acc kernels self + {} + // CHECK-NEXT: acc.kernels { + // CHECK-NEXT: acc.terminator + // CHECK-NEXT: } attributes {selfAttr} + +#pragma acc kernels self(cond) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[COND_LOAD]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.kernels self(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.terminator + // CHECK-NEXT: } loc + +#pragma acc kernels self(0) + {} + // CHECK-NEXT: %[[ZERO_LITERAL:.*]] = cir.const #cir.int<0> : !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[ZERO_LITERAL]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.kernels self(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.terminator + // CHECK-NEXT: } loc + // CHECK-NEXT: cir.return } diff --git a/clang/test/CIR/CodeGenOpenACC/parallel.c b/clang/test/CIR/CodeGenOpenACC/parallel.c index e18270435460c..c7a4bda6faa74 100644 --- a/clang/test/CIR/CodeGenOpenACC/parallel.c +++ b/clang/test/CIR/CodeGenOpenACC/parallel.c @@ -1,7 +1,9 @@ // RUN: %clang_cc1 -fopenacc -emit-cir -fclangir %s -o - | FileCheck %s -void acc_parallel(void) { - // CHECK: cir.func @acc_parallel() { +void acc_parallel(int cond) { + // CHECK: cir.func @acc_parallel(%[[ARG:.*]]: !s32i{{.*}}) { + // CHECK-NEXT: %[[COND:.*]] = cir.alloca !s32i, !cir.ptr, ["cond", init] + // CHECK-NEXT: cir.store %[[ARG]], %[[COND]] : !s32i, !cir.ptr #pragma acc parallel {} // CHECK-NEXT: acc.parallel { @@ -37,5 +39,29 @@ void acc_parallel(void) { // CHECK-NEXT: acc.yield // CHECK-NEXT:} +#pragma acc parallel self + {} + // CHECK-NEXT: acc.parallel { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } attributes {selfAttr} + +#pragma acc parallel self(cond) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[COND_LOAD]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.parallel self(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc parallel self(0) + {} + // CHECK-NEXT: %[[ZERO_LITERAL:.*]] = cir.const #cir.int<0> : !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[ZERO_LITERAL]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.parallel self(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + // CHECK-NEXT: cir.return } diff --git a/clang/test/CIR/CodeGenOpenACC/serial.c b/clang/test/CIR/CodeGenOpenACC/serial.c index 72a0995549da3..38a38ad6c9514 100644 --- a/clang/test/CIR/CodeGenOpenACC/serial.c +++ b/clang/test/CIR/CodeGenOpenACC/serial.c @@ -1,7 +1,9 @@ // RUN: %clang_cc1 -fopenacc -emit-cir -fclangir %s -o - | FileCheck %s -void acc_serial(void) { - // CHECK: cir.func @acc_serial() { +void acc_serial(int cond) { + // CHECK: cir.func @acc_serial(%[[ARG:.*]]: !s32i{{.*}}) { + // CHECK-NEXT: %[[COND:.*]] = cir.alloca !s32i, !cir.ptr, ["cond", init] + // CHECK-NEXT: cir.store %[[ARG]], %[[COND]] : !s32i, !cir.ptr #pragma acc serial {} @@ -38,5 +40,29 @@ void acc_serial(void) { // CHECK-NEXT: acc.yield // CHECK-NEXT:} +#pragma acc serial self + {} + // CHECK-NEXT: acc.serial { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } attributes {selfAttr} + +#pragma acc serial self(cond) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[COND_LOAD]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.serial self(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc serial self(0) + {} + // CHECK-NEXT: %[[ZERO_LITERAL:.*]] = cir.const #cir.int<0> : !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[ZERO_LITERAL]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.serial self(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + // CHECK-NEXT: cir.return } From e4d951d2e42a9124bd87275a864804c4b84b62e3 Mon Sep 17 00:00:00 2001 From: Peter Collingbourne Date: Tue, 15 Apr 2025 16:03:03 -0700 Subject: [PATCH 41/63] LowerTypeTests: Fix quadratic complexity. Currently we have quadratic complexity in LowerTypeTests because ScopedSaveAliaseesAndUsed loops over all aliases for each disjoint set, and the number of aliases and number of disjoint sets is roughly proportional to the program size. Fix that by moving ScopedSaveAliaseesAndUsed to LowerTypeTestsModule::lower() so that we do this only once. Reviewers: fmayer, vitalybuka Reviewed By: vitalybuka Pull Request: https://github.com/llvm/llvm-project/pull/135875 --- llvm/lib/Transforms/IPO/LowerTypeTests.cpp | 164 ++++++++++----------- 1 file changed, 81 insertions(+), 83 deletions(-) diff --git a/llvm/lib/Transforms/IPO/LowerTypeTests.cpp b/llvm/lib/Transforms/IPO/LowerTypeTests.cpp index 7cf7d74acfcfa..8e9f24dfc31fa 100644 --- a/llvm/lib/Transforms/IPO/LowerTypeTests.cpp +++ b/llvm/lib/Transforms/IPO/LowerTypeTests.cpp @@ -1669,61 +1669,55 @@ void LowerTypeTestsModule::buildBitSetsFromFunctionsNative( lowerTypeTestCalls(TypeIds, JumpTable, GlobalLayout); - { - ScopedSaveAliaseesAndUsed S(M); + // Build aliases pointing to offsets into the jump table, and replace + // references to the original functions with references to the aliases. + for (unsigned I = 0; I != Functions.size(); ++I) { + Function *F = cast(Functions[I]->getGlobal()); + bool IsJumpTableCanonical = Functions[I]->isJumpTableCanonical(); - // Build aliases pointing to offsets into the jump table, and replace - // references to the original functions with references to the aliases. - for (unsigned I = 0; I != Functions.size(); ++I) { - Function *F = cast(Functions[I]->getGlobal()); - bool IsJumpTableCanonical = Functions[I]->isJumpTableCanonical(); - - Constant *CombinedGlobalElemPtr = ConstantExpr::getInBoundsGetElementPtr( - JumpTableType, JumpTable, - ArrayRef{ConstantInt::get(IntPtrTy, 0), - ConstantInt::get(IntPtrTy, I)}); - - const bool IsExported = Functions[I]->isExported(); - if (!IsJumpTableCanonical) { - GlobalValue::LinkageTypes LT = IsExported - ? GlobalValue::ExternalLinkage - : GlobalValue::InternalLinkage; - GlobalAlias *JtAlias = GlobalAlias::create(F->getValueType(), 0, LT, - F->getName() + ".cfi_jt", - CombinedGlobalElemPtr, &M); - if (IsExported) - JtAlias->setVisibility(GlobalValue::HiddenVisibility); - else - appendToUsed(M, {JtAlias}); - } + Constant *CombinedGlobalElemPtr = ConstantExpr::getInBoundsGetElementPtr( + JumpTableType, JumpTable, + ArrayRef{ConstantInt::get(IntPtrTy, 0), + ConstantInt::get(IntPtrTy, I)}); + + const bool IsExported = Functions[I]->isExported(); + if (!IsJumpTableCanonical) { + GlobalValue::LinkageTypes LT = IsExported ? GlobalValue::ExternalLinkage + : GlobalValue::InternalLinkage; + GlobalAlias *JtAlias = GlobalAlias::create(F->getValueType(), 0, LT, + F->getName() + ".cfi_jt", + CombinedGlobalElemPtr, &M); + if (IsExported) + JtAlias->setVisibility(GlobalValue::HiddenVisibility); + else + appendToUsed(M, {JtAlias}); + } - if (IsExported) { - if (IsJumpTableCanonical) - ExportSummary->cfiFunctionDefs().emplace(F->getName()); - else - ExportSummary->cfiFunctionDecls().emplace(F->getName()); - } + if (IsExported) { + if (IsJumpTableCanonical) + ExportSummary->cfiFunctionDefs().emplace(F->getName()); + else + ExportSummary->cfiFunctionDecls().emplace(F->getName()); + } - if (!IsJumpTableCanonical) { - if (F->hasExternalWeakLinkage()) - replaceWeakDeclarationWithJumpTablePtr(F, CombinedGlobalElemPtr, - IsJumpTableCanonical); - else - replaceCfiUses(F, CombinedGlobalElemPtr, IsJumpTableCanonical); - } else { - assert(F->getType()->getAddressSpace() == 0); - - GlobalAlias *FAlias = - GlobalAlias::create(F->getValueType(), 0, F->getLinkage(), "", - CombinedGlobalElemPtr, &M); - FAlias->setVisibility(F->getVisibility()); - FAlias->takeName(F); - if (FAlias->hasName()) - F->setName(FAlias->getName() + ".cfi"); - replaceCfiUses(F, FAlias, IsJumpTableCanonical); - if (!F->hasLocalLinkage()) - F->setVisibility(GlobalVariable::HiddenVisibility); - } + if (!IsJumpTableCanonical) { + if (F->hasExternalWeakLinkage()) + replaceWeakDeclarationWithJumpTablePtr(F, CombinedGlobalElemPtr, + IsJumpTableCanonical); + else + replaceCfiUses(F, CombinedGlobalElemPtr, IsJumpTableCanonical); + } else { + assert(F->getType()->getAddressSpace() == 0); + + GlobalAlias *FAlias = GlobalAlias::create( + F->getValueType(), 0, F->getLinkage(), "", CombinedGlobalElemPtr, &M); + FAlias->setVisibility(F->getVisibility()); + FAlias->takeName(F); + if (FAlias->hasName()) + F->setName(FAlias->getName() + ".cfi"); + replaceCfiUses(F, FAlias, IsJumpTableCanonical); + if (!F->hasLocalLinkage()) + F->setVisibility(GlobalVariable::HiddenVisibility); } } @@ -2339,39 +2333,43 @@ bool LowerTypeTestsModule::lower() { if (GlobalClasses.empty()) return false; - // For each disjoint set we found... - for (const auto &C : GlobalClasses) { - if (!C->isLeader()) - continue; - - ++NumTypeIdDisjointSets; - // Build the list of type identifiers in this disjoint set. - std::vector TypeIds; - std::vector Globals; - std::vector ICallBranchFunnels; - for (auto M : GlobalClasses.members(*C)) { - if (isa(M)) - TypeIds.push_back(cast(M)); - else if (isa(M)) - Globals.push_back(cast(M)); - else - ICallBranchFunnels.push_back(cast(M)); - } - - // Order type identifiers by unique ID for determinism. This ordering is - // stable as there is a one-to-one mapping between metadata and unique IDs. - llvm::sort(TypeIds, [&](Metadata *M1, Metadata *M2) { - return TypeIdInfo[M1].UniqueId < TypeIdInfo[M2].UniqueId; - }); + { + ScopedSaveAliaseesAndUsed S(M); + // For each disjoint set we found... + for (const auto &C : GlobalClasses) { + if (!C->isLeader()) + continue; - // Same for the branch funnels. - llvm::sort(ICallBranchFunnels, - [&](ICallBranchFunnel *F1, ICallBranchFunnel *F2) { - return F1->UniqueId < F2->UniqueId; - }); + ++NumTypeIdDisjointSets; + // Build the list of type identifiers in this disjoint set. + std::vector TypeIds; + std::vector Globals; + std::vector ICallBranchFunnels; + for (auto M : GlobalClasses.members(*C)) { + if (isa(M)) + TypeIds.push_back(cast(M)); + else if (isa(M)) + Globals.push_back(cast(M)); + else + ICallBranchFunnels.push_back(cast(M)); + } - // Build bitsets for this disjoint set. - buildBitSetsFromDisjointSet(TypeIds, Globals, ICallBranchFunnels); + // Order type identifiers by unique ID for determinism. This ordering is + // stable as there is a one-to-one mapping between metadata and unique + // IDs. + llvm::sort(TypeIds, [&](Metadata *M1, Metadata *M2) { + return TypeIdInfo[M1].UniqueId < TypeIdInfo[M2].UniqueId; + }); + + // Same for the branch funnels. + llvm::sort(ICallBranchFunnels, + [&](ICallBranchFunnel *F1, ICallBranchFunnel *F2) { + return F1->UniqueId < F2->UniqueId; + }); + + // Build bitsets for this disjoint set. + buildBitSetsFromDisjointSet(TypeIds, Globals, ICallBranchFunnels); + } } allocateByteArrays(); From 3428cc94c893f9a09728c707baf018b9cdfaf243 Mon Sep 17 00:00:00 2001 From: Eugene Epshteyn Date: Tue, 15 Apr 2025 19:04:59 -0400 Subject: [PATCH 42/63] [flang] Implement external routine usage of hostnm() (#134900) Previously, `hostnm` extended intrinsic was implemented as proper intrinsic. Since then we found out that some applications use `hostnm` as external routine via `external hostnm`. This prevents `hostnm` from being recognized as an intrinsic. This PR implements `hostnm` as external routine. --- flang-rt/lib/runtime/extensions.cpp | 32 ++++++++++++++++++++++++ flang/include/flang/Runtime/extensions.h | 2 +- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/flang-rt/lib/runtime/extensions.cpp b/flang-rt/lib/runtime/extensions.cpp index 6b553ff97e5ab..e70dff3997233 100644 --- a/flang-rt/lib/runtime/extensions.cpp +++ b/flang-rt/lib/runtime/extensions.cpp @@ -264,6 +264,38 @@ int RTNAME(Chdir)(const char *name) { #endif } +int FORTRAN_PROCEDURE_NAME(hostnm)(char *hn, int length) { + std::int32_t status{0}; + + if (!hn || length < 0) { + return EINVAL; + } + +#ifdef _WIN32 + DWORD dwSize{static_cast(length)}; + + // Note: Winsock has gethostname(), but use Win32 API GetComputerNameEx(), + // in order to avoid adding dependency on Winsock. + if (!GetComputerNameExA(ComputerNameDnsHostname, hn, &dwSize)) { + status = GetLastError(); + } +#else + if (gethostname(hn, length) < 0) { + status = errno; + } +#endif + + if (status == 0) { + // Find zero terminator and fill the string from the + // zero terminator to the end with spaces + char *str_end{hn + length}; + char *str_zero{std::find(hn, str_end, '\0')}; + std::fill(str_zero, str_end, ' '); + } + + return status; +} + int FORTRAN_PROCEDURE_NAME(ierrno)() { return errno; } void FORTRAN_PROCEDURE_NAME(qsort)(int *array, int *len, int *isize, diff --git a/flang/include/flang/Runtime/extensions.h b/flang/include/flang/Runtime/extensions.h index db2245875e85a..06ae7f35d9b5b 100644 --- a/flang/include/flang/Runtime/extensions.h +++ b/flang/include/flang/Runtime/extensions.h @@ -60,7 +60,7 @@ uid_t RTNAME(GetUID)(); void FORTRAN_PROCEDURE_NAME(getlog)(char *name, std::int64_t length); // GNU extension subroutine HOSTNM(C) -void FORTRAN_PROCEDURE_NAME(hostnm)(char *name, std::int64_t length); +int FORTRAN_PROCEDURE_NAME(hostnm)(char *hn, int length); std::intptr_t RTNAME(Malloc)(std::size_t size); From 58c3fba7063eaca926931a412c329e9ac4deefd6 Mon Sep 17 00:00:00 2001 From: Peter Collingbourne Date: Tue, 15 Apr 2025 16:20:45 -0700 Subject: [PATCH 43/63] Revert "LowerTypeTests: Fix quadratic complexity." This reverts commit e4d951d2e42a9124bd87275a864804c4b84b62e3. Need to investigate some test failures. --- llvm/lib/Transforms/IPO/LowerTypeTests.cpp | 164 +++++++++++---------- 1 file changed, 83 insertions(+), 81 deletions(-) diff --git a/llvm/lib/Transforms/IPO/LowerTypeTests.cpp b/llvm/lib/Transforms/IPO/LowerTypeTests.cpp index 8e9f24dfc31fa..7cf7d74acfcfa 100644 --- a/llvm/lib/Transforms/IPO/LowerTypeTests.cpp +++ b/llvm/lib/Transforms/IPO/LowerTypeTests.cpp @@ -1669,55 +1669,61 @@ void LowerTypeTestsModule::buildBitSetsFromFunctionsNative( lowerTypeTestCalls(TypeIds, JumpTable, GlobalLayout); - // Build aliases pointing to offsets into the jump table, and replace - // references to the original functions with references to the aliases. - for (unsigned I = 0; I != Functions.size(); ++I) { - Function *F = cast(Functions[I]->getGlobal()); - bool IsJumpTableCanonical = Functions[I]->isJumpTableCanonical(); + { + ScopedSaveAliaseesAndUsed S(M); - Constant *CombinedGlobalElemPtr = ConstantExpr::getInBoundsGetElementPtr( - JumpTableType, JumpTable, - ArrayRef{ConstantInt::get(IntPtrTy, 0), - ConstantInt::get(IntPtrTy, I)}); - - const bool IsExported = Functions[I]->isExported(); - if (!IsJumpTableCanonical) { - GlobalValue::LinkageTypes LT = IsExported ? GlobalValue::ExternalLinkage - : GlobalValue::InternalLinkage; - GlobalAlias *JtAlias = GlobalAlias::create(F->getValueType(), 0, LT, - F->getName() + ".cfi_jt", - CombinedGlobalElemPtr, &M); - if (IsExported) - JtAlias->setVisibility(GlobalValue::HiddenVisibility); - else - appendToUsed(M, {JtAlias}); - } + // Build aliases pointing to offsets into the jump table, and replace + // references to the original functions with references to the aliases. + for (unsigned I = 0; I != Functions.size(); ++I) { + Function *F = cast(Functions[I]->getGlobal()); + bool IsJumpTableCanonical = Functions[I]->isJumpTableCanonical(); + + Constant *CombinedGlobalElemPtr = ConstantExpr::getInBoundsGetElementPtr( + JumpTableType, JumpTable, + ArrayRef{ConstantInt::get(IntPtrTy, 0), + ConstantInt::get(IntPtrTy, I)}); + + const bool IsExported = Functions[I]->isExported(); + if (!IsJumpTableCanonical) { + GlobalValue::LinkageTypes LT = IsExported + ? GlobalValue::ExternalLinkage + : GlobalValue::InternalLinkage; + GlobalAlias *JtAlias = GlobalAlias::create(F->getValueType(), 0, LT, + F->getName() + ".cfi_jt", + CombinedGlobalElemPtr, &M); + if (IsExported) + JtAlias->setVisibility(GlobalValue::HiddenVisibility); + else + appendToUsed(M, {JtAlias}); + } - if (IsExported) { - if (IsJumpTableCanonical) - ExportSummary->cfiFunctionDefs().emplace(F->getName()); - else - ExportSummary->cfiFunctionDecls().emplace(F->getName()); - } + if (IsExported) { + if (IsJumpTableCanonical) + ExportSummary->cfiFunctionDefs().emplace(F->getName()); + else + ExportSummary->cfiFunctionDecls().emplace(F->getName()); + } - if (!IsJumpTableCanonical) { - if (F->hasExternalWeakLinkage()) - replaceWeakDeclarationWithJumpTablePtr(F, CombinedGlobalElemPtr, - IsJumpTableCanonical); - else - replaceCfiUses(F, CombinedGlobalElemPtr, IsJumpTableCanonical); - } else { - assert(F->getType()->getAddressSpace() == 0); - - GlobalAlias *FAlias = GlobalAlias::create( - F->getValueType(), 0, F->getLinkage(), "", CombinedGlobalElemPtr, &M); - FAlias->setVisibility(F->getVisibility()); - FAlias->takeName(F); - if (FAlias->hasName()) - F->setName(FAlias->getName() + ".cfi"); - replaceCfiUses(F, FAlias, IsJumpTableCanonical); - if (!F->hasLocalLinkage()) - F->setVisibility(GlobalVariable::HiddenVisibility); + if (!IsJumpTableCanonical) { + if (F->hasExternalWeakLinkage()) + replaceWeakDeclarationWithJumpTablePtr(F, CombinedGlobalElemPtr, + IsJumpTableCanonical); + else + replaceCfiUses(F, CombinedGlobalElemPtr, IsJumpTableCanonical); + } else { + assert(F->getType()->getAddressSpace() == 0); + + GlobalAlias *FAlias = + GlobalAlias::create(F->getValueType(), 0, F->getLinkage(), "", + CombinedGlobalElemPtr, &M); + FAlias->setVisibility(F->getVisibility()); + FAlias->takeName(F); + if (FAlias->hasName()) + F->setName(FAlias->getName() + ".cfi"); + replaceCfiUses(F, FAlias, IsJumpTableCanonical); + if (!F->hasLocalLinkage()) + F->setVisibility(GlobalVariable::HiddenVisibility); + } } } @@ -2333,43 +2339,39 @@ bool LowerTypeTestsModule::lower() { if (GlobalClasses.empty()) return false; - { - ScopedSaveAliaseesAndUsed S(M); - // For each disjoint set we found... - for (const auto &C : GlobalClasses) { - if (!C->isLeader()) - continue; - - ++NumTypeIdDisjointSets; - // Build the list of type identifiers in this disjoint set. - std::vector TypeIds; - std::vector Globals; - std::vector ICallBranchFunnels; - for (auto M : GlobalClasses.members(*C)) { - if (isa(M)) - TypeIds.push_back(cast(M)); - else if (isa(M)) - Globals.push_back(cast(M)); - else - ICallBranchFunnels.push_back(cast(M)); - } + // For each disjoint set we found... + for (const auto &C : GlobalClasses) { + if (!C->isLeader()) + continue; - // Order type identifiers by unique ID for determinism. This ordering is - // stable as there is a one-to-one mapping between metadata and unique - // IDs. - llvm::sort(TypeIds, [&](Metadata *M1, Metadata *M2) { - return TypeIdInfo[M1].UniqueId < TypeIdInfo[M2].UniqueId; - }); - - // Same for the branch funnels. - llvm::sort(ICallBranchFunnels, - [&](ICallBranchFunnel *F1, ICallBranchFunnel *F2) { - return F1->UniqueId < F2->UniqueId; - }); - - // Build bitsets for this disjoint set. - buildBitSetsFromDisjointSet(TypeIds, Globals, ICallBranchFunnels); + ++NumTypeIdDisjointSets; + // Build the list of type identifiers in this disjoint set. + std::vector TypeIds; + std::vector Globals; + std::vector ICallBranchFunnels; + for (auto M : GlobalClasses.members(*C)) { + if (isa(M)) + TypeIds.push_back(cast(M)); + else if (isa(M)) + Globals.push_back(cast(M)); + else + ICallBranchFunnels.push_back(cast(M)); } + + // Order type identifiers by unique ID for determinism. This ordering is + // stable as there is a one-to-one mapping between metadata and unique IDs. + llvm::sort(TypeIds, [&](Metadata *M1, Metadata *M2) { + return TypeIdInfo[M1].UniqueId < TypeIdInfo[M2].UniqueId; + }); + + // Same for the branch funnels. + llvm::sort(ICallBranchFunnels, + [&](ICallBranchFunnel *F1, ICallBranchFunnel *F2) { + return F1->UniqueId < F2->UniqueId; + }); + + // Build bitsets for this disjoint set. + buildBitSetsFromDisjointSet(TypeIds, Globals, ICallBranchFunnels); } allocateByteArrays(); From 860d0383db80ea881e957a5628d04e9d725b919d Mon Sep 17 00:00:00 2001 From: Haowei Date: Tue, 15 Apr 2025 16:23:53 -0700 Subject: [PATCH 44/63] [Fuchsia] Not building llvm-mt when LIBXML2 is not enabled. (#135877) This patch prevents including the llvm-mt to `LLVM_TOOLCHAIN_TOOLS` in the Fuchsia toolchain when LIBXML2 is not explicitly enabled. --- clang/cmake/caches/Fuchsia-stage2.cmake | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/clang/cmake/caches/Fuchsia-stage2.cmake b/clang/cmake/caches/Fuchsia-stage2.cmake index 99890b8246ad7..e10855f5ef31b 100644 --- a/clang/cmake/caches/Fuchsia-stage2.cmake +++ b/clang/cmake/caches/Fuchsia-stage2.cmake @@ -459,7 +459,6 @@ set(LLVM_TOOLCHAIN_TOOLS llvm-libtool-darwin llvm-lipo llvm-ml - llvm-mt llvm-nm llvm-objcopy llvm-objdump @@ -481,6 +480,10 @@ set(LLVM_TOOLCHAIN_TOOLS scan-build-py CACHE STRING "") +if (LLVM_ENABLE_LIBXML2) + list(APPEND LLVM_TOOLCHAIN_TOOLS llvm-mt) +endif() + set(LLVM_Toolchain_DISTRIBUTION_COMPONENTS bolt clang From 6ad922b75a41911e0e394d5d367bee1240ad509f Mon Sep 17 00:00:00 2001 From: erichkeane Date: Tue, 15 Apr 2025 16:14:49 -0700 Subject: [PATCH 45/63] [OpenACC][CIR] Implement lowering for 'if' on compute constructs This is the same for these as the 'self' was, except it doesn't support the 'empty' variant, so we have to just generate the condition. This patch does that, and extracts the 'condition' emission to a separate function since the two share it. --- clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp | 40 +++++++++++++----- clang/test/CIR/CodeGenOpenACC/kernels.c | 46 ++++++++++++++++++++- clang/test/CIR/CodeGenOpenACC/parallel.c | 44 +++++++++++++++++++- clang/test/CIR/CodeGenOpenACC/serial.c | 44 +++++++++++++++++++- 4 files changed, 159 insertions(+), 15 deletions(-) diff --git a/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp b/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp index 3bcc6f908a841..c14ff9a16841d 100644 --- a/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp +++ b/clang/lib/CIR/CodeGen/CIRGenStmtOpenACC.cpp @@ -50,6 +50,21 @@ class OpenACCClauseCIREmitter final cgf.cgm.errorNYI(c.getSourceRange(), "OpenACC Clause", c.getClauseKind()); } + // 'condition' as an OpenACC grammar production is used for 'if' and (some + // variants of) 'self'. It needs to be emitted as a signless-1-bit value, so + // this function emits the expression, then sets the unrealized conversion + // cast correctly, and returns the completed value. + mlir::Value createCondition(const Expr *condExpr) { + mlir::Value condition = cgf.evaluateExprAsBool(condExpr); + mlir::Location exprLoc = cgf.cgm.getLoc(condExpr->getBeginLoc()); + mlir::IntegerType targetType = mlir::IntegerType::get( + &cgf.getMLIRContext(), /*width=*/1, + mlir::IntegerType::SignednessSemantics::Signless); + auto conversionOp = builder.create( + exprLoc, targetType, condition); + return conversionOp.getResult(0); + } + public: OpenACCClauseCIREmitter(OpTy &operation, CIRGenFunction &cgf, CIRGenBuilderTy &builder, @@ -132,17 +147,8 @@ class OpenACCClauseCIREmitter final operation.setSelfAttr(true); } else if (clause.isConditionExprClause()) { assert(clause.hasConditionExpr()); - mlir::Value condition = - cgf.evaluateExprAsBool(clause.getConditionExpr()); - - mlir::Location exprLoc = - cgf.cgm.getLoc(clause.getConditionExpr()->getBeginLoc()); - mlir::IntegerType targetType = mlir::IntegerType::get( - &cgf.getMLIRContext(), /*width=*/1, - mlir::IntegerType::SignednessSemantics::Signless); - auto conversionOp = builder.create( - exprLoc, targetType, condition); - operation.getSelfCondMutable().append(conversionOp.getResult(0)); + operation.getSelfCondMutable().append( + createCondition(clause.getConditionExpr())); } else { llvm_unreachable("var-list version of self shouldn't get here"); } @@ -150,6 +156,18 @@ class OpenACCClauseCIREmitter final return clauseNotImplemented(clause); } } + + void VisitIfClause(const OpenACCIfClause &clause) { + if constexpr (isOneOfTypes) { + operation.getIfCondMutable().append( + createCondition(clause.getConditionExpr())); + } else { + // 'if' applies to most of the constructs, but hold off on lowering them + // until we can write tests/know what we're doing with codegen to make + // sure we get it right. + return clauseNotImplemented(clause); + } + } }; template diff --git a/clang/test/CIR/CodeGenOpenACC/kernels.c b/clang/test/CIR/CodeGenOpenACC/kernels.c index 934daf9e8ecc0..ca5bfebcb4ff3 100644 --- a/clang/test/CIR/CodeGenOpenACC/kernels.c +++ b/clang/test/CIR/CodeGenOpenACC/kernels.c @@ -1,4 +1,4 @@ -// RUN: %clang_cc1 -fopenacc -emit-cir -fclangir %s -o - | FileCheck %s +// RUN: %clang_cc1 -fopenacc -Wno-openacc-self-if-potential-conflict -emit-cir -fclangir %s -o - | FileCheck %s void acc_kernels(int cond) { // CHECK: cir.func @acc_kernels(%[[ARG:.*]]: !s32i{{.*}}) { @@ -63,6 +63,48 @@ void acc_kernels(int cond) { // CHECK-NEXT: acc.kernels self(%[[CONV_CAST]]) { // CHECK-NEXT: acc.terminator // CHECK-NEXT: } loc - + +#pragma acc kernels if(cond) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[COND_LOAD]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.kernels if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.terminator + // CHECK-NEXT: } loc + +#pragma acc kernels if(1) + {} + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[ONE_LITERAL]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.kernels if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.terminator + // CHECK-NEXT: } loc + +#pragma acc kernels if(cond == 1) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[EQ_RES:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[ONE_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES]] : !cir.bool to i1 + // CHECK-NEXT: acc.kernels if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.terminator + // CHECK-NEXT: } loc + +#pragma acc kernels if(cond == 1) self(cond == 2) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[EQ_RES_IF:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[ONE_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST_IF:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES_IF]] : !cir.bool to i1 + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[TWO_LITERAL:.*]] = cir.const #cir.int<2> : !s32i + // CHECK-NEXT: %[[EQ_RES_SELF:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[TWO_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST_SELF:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES_SELF]] : !cir.bool to i1 + // CHECK-NEXT: acc.kernels self(%[[CONV_CAST_SELF]]) if(%[[CONV_CAST_IF]]) { + // CHECK-NEXT: acc.terminator + // CHECK-NEXT: } loc + // CHECK-NEXT: cir.return } diff --git a/clang/test/CIR/CodeGenOpenACC/parallel.c b/clang/test/CIR/CodeGenOpenACC/parallel.c index c7a4bda6faa74..3fb0b987409db 100644 --- a/clang/test/CIR/CodeGenOpenACC/parallel.c +++ b/clang/test/CIR/CodeGenOpenACC/parallel.c @@ -1,4 +1,4 @@ -// RUN: %clang_cc1 -fopenacc -emit-cir -fclangir %s -o - | FileCheck %s +// RUN: %clang_cc1 -fopenacc -Wno-openacc-self-if-potential-conflict -emit-cir -fclangir %s -o - | FileCheck %s void acc_parallel(int cond) { // CHECK: cir.func @acc_parallel(%[[ARG:.*]]: !s32i{{.*}}) { @@ -63,5 +63,47 @@ void acc_parallel(int cond) { // CHECK-NEXT: acc.yield // CHECK-NEXT: } loc +#pragma acc parallel if(cond) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[COND_LOAD]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.parallel if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc parallel if(1) + {} + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[ONE_LITERAL]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.parallel if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc parallel if(cond == 1) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[EQ_RES:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[ONE_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES]] : !cir.bool to i1 + // CHECK-NEXT: acc.parallel if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc parallel if(cond == 1) self(cond == 2) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[EQ_RES_IF:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[ONE_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST_IF:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES_IF]] : !cir.bool to i1 + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[TWO_LITERAL:.*]] = cir.const #cir.int<2> : !s32i + // CHECK-NEXT: %[[EQ_RES_SELF:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[TWO_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST_SELF:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES_SELF]] : !cir.bool to i1 + // CHECK-NEXT: acc.parallel self(%[[CONV_CAST_SELF]]) if(%[[CONV_CAST_IF]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + // CHECK-NEXT: cir.return } diff --git a/clang/test/CIR/CodeGenOpenACC/serial.c b/clang/test/CIR/CodeGenOpenACC/serial.c index 38a38ad6c9514..b72f44a2ea473 100644 --- a/clang/test/CIR/CodeGenOpenACC/serial.c +++ b/clang/test/CIR/CodeGenOpenACC/serial.c @@ -1,4 +1,4 @@ -// RUN: %clang_cc1 -fopenacc -emit-cir -fclangir %s -o - | FileCheck %s +// RUN: %clang_cc1 -fopenacc -Wno-openacc-self-if-potential-conflict -emit-cir -fclangir %s -o - | FileCheck %s void acc_serial(int cond) { // CHECK: cir.func @acc_serial(%[[ARG:.*]]: !s32i{{.*}}) { @@ -64,5 +64,47 @@ void acc_serial(int cond) { // CHECK-NEXT: acc.yield // CHECK-NEXT: } loc +#pragma acc serial if(cond) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[COND_LOAD]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.serial if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc serial if(1) + {} + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[BOOL_CAST:.*]] = cir.cast(int_to_bool, %[[ONE_LITERAL]] : !s32i), !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[BOOL_CAST]] : !cir.bool to i1 + // CHECK-NEXT: acc.serial if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc serial if(cond == 1) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[EQ_RES:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[ONE_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES]] : !cir.bool to i1 + // CHECK-NEXT: acc.serial if(%[[CONV_CAST]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + +#pragma acc serial if(cond == 1) self(cond == 2) + {} + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[ONE_LITERAL:.*]] = cir.const #cir.int<1> : !s32i + // CHECK-NEXT: %[[EQ_RES_IF:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[ONE_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST_IF:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES_IF]] : !cir.bool to i1 + // CHECK-NEXT: %[[COND_LOAD:.*]] = cir.load %[[COND]] : !cir.ptr, !s32i + // CHECK-NEXT: %[[TWO_LITERAL:.*]] = cir.const #cir.int<2> : !s32i + // CHECK-NEXT: %[[EQ_RES_SELF:.*]] = cir.cmp(eq, %[[COND_LOAD]], %[[TWO_LITERAL]]) : !s32i, !cir.bool + // CHECK-NEXT: %[[CONV_CAST_SELF:.*]] = builtin.unrealized_conversion_cast %[[EQ_RES_SELF]] : !cir.bool to i1 + // CHECK-NEXT: acc.serial self(%[[CONV_CAST_SELF]]) if(%[[CONV_CAST_IF]]) { + // CHECK-NEXT: acc.yield + // CHECK-NEXT: } loc + // CHECK-NEXT: cir.return } From 0b8f817aab7a242e0bfb519cb07c8979ffadef36 Mon Sep 17 00:00:00 2001 From: Maksim Panchenko Date: Tue, 15 Apr 2025 16:59:05 -0700 Subject: [PATCH 46/63] [BOLT] Fix conditional compilation of hugify.cpp (#135880) Fix builds after #117158: do not build hugify.cpp on Apple platforms. --- bolt/runtime/hugify.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/bolt/runtime/hugify.cpp b/bolt/runtime/hugify.cpp index 67d5fa26007d2..672b04247dfa4 100644 --- a/bolt/runtime/hugify.cpp +++ b/bolt/runtime/hugify.cpp @@ -6,8 +6,8 @@ // //===---------------------------------------------------------------------===// -#if defined(__x86_64__) || \ - (defined(__aarch64__) || defined(__arm64__)) && !defined(__APPLE__) +#if (defined(__x86_64__) || defined(__aarch64__) || defined(__arm64__)) && \ + !defined(__APPLE__) #include "common.h" From bd4d3519c708d70ed8c827a27b63f13b0229ef00 Mon Sep 17 00:00:00 2001 From: Camsyn Date: Wed, 16 Apr 2025 08:15:57 +0800 Subject: [PATCH 47/63] [ASan] Prevent ASan/LSan deadlock by preloading modules before error reporting (#131756) ### Description This PR resolves a deadlock between AddressSanitizer (ASan) and LeakSanitizer (LSan) that occurs when both sanitizers attempt to acquire locks in conflicting orders across threads. The fix ensures safe lock acquisition ordering by preloading module information before error reporting. --- ### Issue Details **Reproducer** ```cpp // Thread 1: ASan error path int arr[1] = {0}; std::thread t([&]() { arr[1] = 1; // Triggers ASan OOB error }); // Thread 2: LSan check path __lsan_do_leak_check(); ``` **Lock Order Conflict**: - Thread 1 (ASan error reporting): 1. Acquires ASan thread registry lock (B) 1. Attempts to acquire libdl lock (A) via `dl_iterate_phdr` - Thread 2 (LSan leak check): 1. Acquires libdl lock (A) via `dl_iterate_phdr` 1. Attempts to acquire ASan thread registry lock (B) This creates a circular wait condition (A -> B -> A) meeting all four Coffman deadlock criteria. --- ### Fix Strategy The root cause lies in ASan's error reporting path needing `dl_iterate_phdr` (requiring lock A) while already holding its thread registry lock (B). The solution: 1. **Preload Modules Early**: Force module list initialization _before_ acquiring ASan's thread lock 2. **Avoid Nested Locking**: Ensure symbolization (via dl_iterate_phdr) completes before error reporting locks Key code change: ```cpp // Before acquiring ASan's thread registry lock: Symbolizer::GetOrInit()->GetRefreshedListOfModules(); ``` This guarantees module information is cached before lock acquisition, eliminating the need for `dl_iterate_phdr` calls during error reporting. --- ### Testing Added **asan_lsan_deadlock.cpp** test case: - Reproduces deadlock reliably without fix **under idle system conditions** - Uses watchdog thread to detect hangs - Verifies ASan error reports correctly without deadlock **Note**: Due to the inherent non-determinism of thread scheduling and lock acquisition timing, this test may not reliably reproduce the deadlock on busy systems (e.g., during parallel `ninja check-asan` runs). --- ### Impact - Fixes rare but severe deadlocks in mixed ASan+LSan environments - Maintains thread safety guarantees for both sanitizers - No user-visible behavior changes except deadlock elimination --- ### Relevant Buggy Code - Code in ASan's asan_report.cpp ```cpp explicit ScopedInErrorReport(bool fatal = false) : halt_on_error_(fatal || flags()->halt_on_error) { // Acquire lock B asanThreadRegistry().Lock(); } ~ScopedInErrorReport() { ... // Try to acquire lock A under holding lock B via the following path // #4 0x000071a353d83e93 in __GI___dl_iterate_phdr ( // callback=0x5d1a07a39580 <__sanitizer::dl_iterate_phdr_cb(dl_phdr_info*, unsigned long, void*)>, // data=0x6da3510fd3f0) at ./elf/dl-iteratephdr.c:39 // #5 0x00005d1a07a39574 in __sanitizer::ListOfModules::init (this=0x71a353ebc080) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:784 // #6 0x00005d1a07a429e3 in __sanitizer::Symbolizer::RefreshModules (this=0x71a353ebc058) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:188 // #7 __sanitizer::Symbolizer::FindModuleForAddress (this=this@entry=0x71a353ebc058, // address=address@entry=102366378805727) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:214 // #8 0x00005d1a07a4291b in __sanitizer::Symbolizer::SymbolizePC (this=0x71a353ebc058, addr=102366378805727) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:88 // #9 0x00005d1a07a40df7 in __sanitizer::(anonymous namespace)::StackTraceTextPrinter::ProcessAddressFrames ( // this=this@entry=0x6da3510fd520, pc=102366378805727) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:37 // #10 0x00005d1a07a40d27 in __sanitizer::StackTrace::PrintTo (this=this@entry=0x6da3510fd5e8, // output=output@entry=0x6da3510fd588) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:110 // #11 0x00005d1a07a410a1 in __sanitizer::StackTrace::Print (this=0x6da3510fd5e8) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:133 // #12 0x00005d1a0798758d in __asan::ErrorGeneric::Print ( // this=0x5d1a07aa4e08 <__asan::ScopedInErrorReport::current_error_+8>) // at llvm-project/compiler-rt/lib/asan/asan_errors.cpp:617 current_error_.Print(); ... } ``` - Code in LSan's lsan_common_linux.cpp ```cpp void LockStuffAndStopTheWorld(StopTheWorldCallback callback, CheckForLeaksParam *argument) { // Acquire lock A dl_iterate_phdr(LockStuffAndStopTheWorldCallback, ¶m); } static int LockStuffAndStopTheWorldCallback(struct dl_phdr_info *info, size_t size, void *data) { // Try to acquire lock B under holding lock A via the following path // #3 0x000055555562b34a in __sanitizer::ThreadRegistry::Lock (this=) // at llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:99 // #4 __lsan::LockThreads () at llvm-project/compiler-rt/lib/asan/asan_thread.cpp:484 // #5 0x0000555555652629 in __lsan::ScopedStopTheWorldLock::ScopedStopTheWorldLock (this=) // at llvm-project/compiler-rt/lib/lsan/lsan_common.h:164 // #6 __lsan::LockStuffAndStopTheWorldCallback (info=, size=, data=0x0, // data@entry=0x7fffffffd158) at llvm-project/compiler-rt/lib/lsan/lsan_common_linux.cpp:120 ScopedStopTheWorldLock lock; DoStopTheWorldParam *param = reinterpret_cast(data); StopTheWorld(param->callback, param->argument); return 1; } ``` --- compiler-rt/lib/asan/asan_report.cpp | 25 +++++++ .../asan/TestCases/asan_lsan_deadlock.cpp | 72 +++++++++++++++++++ 2 files changed, 97 insertions(+) create mode 100644 compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp diff --git a/compiler-rt/lib/asan/asan_report.cpp b/compiler-rt/lib/asan/asan_report.cpp index 6302866805f37..e515f20548c00 100644 --- a/compiler-rt/lib/asan/asan_report.cpp +++ b/compiler-rt/lib/asan/asan_report.cpp @@ -126,6 +126,31 @@ class ScopedInErrorReport { public: explicit ScopedInErrorReport(bool fatal = false) : halt_on_error_(fatal || flags()->halt_on_error) { + // Deadlock Prevention Between ASan and LSan + // + // Background: + // - The `dl_iterate_phdr` function requires holding libdl's internal lock + // (Lock A). + // - LSan acquires the ASan thread registry lock (Lock B) *after* calling + // `dl_iterate_phdr`. + // + // Problem Scenario: + // When ASan attempts to call `dl_iterate_phdr` while holding Lock B (e.g., + // during error reporting via `ErrorDescription::Print`), a circular lock + // dependency may occur: + // 1. Thread 1: Holds Lock B → Requests Lock A (via dl_iterate_phdr) + // 2. Thread 2: Holds Lock A → Requests Lock B (via LSan operations) + // + // Solution: + // Proactively load all required modules before acquiring Lock B. + // This ensures: + // 1. Any `dl_iterate_phdr` calls during module loading complete before + // locking. + // 2. Subsequent error reporting avoids nested lock acquisition patterns. + // 3. Eliminates the lock order inversion risk between libdl and ASan's + // thread registry. + Symbolizer::GetOrInit()->GetRefreshedListOfModules(); + // Make sure the registry and sanitizer report mutexes are locked while // we're printing an error report. // We can lock them only here to avoid self-deadlock in case of diff --git a/compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp b/compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp new file mode 100644 index 0000000000000..4e1a2415ad013 --- /dev/null +++ b/compiler-rt/test/asan/TestCases/asan_lsan_deadlock.cpp @@ -0,0 +1,72 @@ +// Test for potential deadlock in LeakSanitizer+AddressSanitizer. +// REQUIRES: leak-detection +// +// RUN: %clangxx_asan -O0 %s -o %t +// RUN: %env_asan_opts=detect_leaks=1 not %run %t 2>&1 | FileCheck %s + +/* + * Purpose: Verify deadlock prevention between ASan error reporting and LSan leak checking. + * + * Test Design: + * 1. Creates contention scenario between: + * - ASan's error reporting (requires lock B -> lock A ordering) + * - LSan's leak check (requires lock A -> lock B ordering) + * 2. Thread timing: + * - Main thread: Holds 'in' mutex -> Triggers LSan check (lock A then B) + * - Worker thread: Triggers ASan OOB error (lock B then A via symbolization) + * + * Deadlock Condition (if unfixed): + * Circular lock dependency forms when: + * [Main Thread] LSan: lock A -> requests lock B + * [Worker Thread] ASan: lock B -> requests lock A + * + * Success Criteria: + * With proper lock ordering enforcement, watchdog should NOT trigger - test exits normally. + * If deadlock occurs, watchdog terminates via _exit(1) after 10s timeout. + */ + +#include +#include +#include +#include +#include + +void Watchdog() { + // Safety mechanism: Turn infinite deadlock into finite test failure + usleep(10000000); + // CHECK-NOT: Timeout! Deadlock detected. + puts("Timeout! Deadlock detected."); + fflush(stdout); + _exit(1); +} + +int main(int argc, char **argv) { + int arr[1] = {0}; + std::mutex in; + in.lock(); + + std::thread w(Watchdog); + w.detach(); + + std::thread t([&]() { + in.unlock(); + /* + * Provoke ASan error: ASan's error reporting acquires: + * 1. ASan's thread registry lock (B) during the reporting + * 2. dl_iterate_phdr lock (A) during symbolization + */ + // CHECK: SUMMARY: AddressSanitizer: stack-buffer-overflow + arr[argc] = 1; // Deliberate OOB access + }); + + in.lock(); + /* + * Critical section: LSan's check acquires: + * 1. dl_iterate_phdr lock (A) + * 2. ASan's thread registry lock (B) + * before Stop The World. + */ + __lsan_do_leak_check(); + t.join(); + return 0; +} From 77f0708b9d4feee8b8a67a5f571be741be4e26af Mon Sep 17 00:00:00 2001 From: Vinay Deshmukh <32487576+vinay-deshmukh@users.noreply.github.com> Date: Tue, 15 Apr 2025 20:24:07 -0400 Subject: [PATCH 48/63] [libc]: Remove `-Wglobal-constructors` for libc tests (#131485) * Relates to: https://github.com/llvm/llvm-project/issues/119281 * Removes `-Wglobal-constructors` as per: https://github.com/llvm/llvm-project/pull/131485#pullrequestreview-2728020622 --- libc/cmake/modules/LLVMLibCTestRules.cmake | 1 - 1 file changed, 1 deletion(-) diff --git a/libc/cmake/modules/LLVMLibCTestRules.cmake b/libc/cmake/modules/LLVMLibCTestRules.cmake index 63a8e9ecda002..a28e15fc5e394 100644 --- a/libc/cmake/modules/LLVMLibCTestRules.cmake +++ b/libc/cmake/modules/LLVMLibCTestRules.cmake @@ -57,7 +57,6 @@ function(_get_common_test_compile_options output_var c_test flags) list(APPEND compile_options "-Wnewline-eof") list(APPEND compile_options "-Wnonportable-system-include-path") list(APPEND compile_options "-Wthread-safety") - # list(APPEND compile_options "-Wglobal-constructors") endif() endif() set(${output_var} ${compile_options} PARENT_SCOPE) From 6c6ab2a270b799f1397926c9064fa30fe2be1d96 Mon Sep 17 00:00:00 2001 From: YunQiang Su Date: Wed, 16 Apr 2025 09:09:13 +0800 Subject: [PATCH 49/63] AArch64: Set FMAXIMUMNUM and FMINIMUMNUM as Promote if not fullfp16 (#135708) Since Promote will emit FP_EXTEND, the result of it will never be sNaN, so we don't need worry about duplicated of FCANONICALIZE in expandFMINIMUMNUM_FMAXIMUMNUM. --- .../Target/AArch64/AArch64ISelLowering.cpp | 2 + .../AArch64/fp-maximumnum-minimumnum.ll | 1751 ++++++++++++++--- 2 files changed, 1480 insertions(+), 273 deletions(-) diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp index 830ec6886e6bc..bea8087750d6e 100644 --- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp +++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp @@ -795,6 +795,8 @@ AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM, ISD::FMAXNUM, ISD::FMINIMUM, ISD::FMAXIMUM, + ISD::FMINIMUMNUM, + ISD::FMAXIMUMNUM, ISD::FCANONICALIZE, ISD::STRICT_FADD, ISD::STRICT_FSUB, diff --git a/llvm/test/CodeGen/AArch64/fp-maximumnum-minimumnum.ll b/llvm/test/CodeGen/AArch64/fp-maximumnum-minimumnum.ll index bb3f9a3e52a16..c6b8e41f9bdfd 100644 --- a/llvm/test/CodeGen/AArch64/fp-maximumnum-minimumnum.ll +++ b/llvm/test/CodeGen/AArch64/fp-maximumnum-minimumnum.ll @@ -1,5 +1,6 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py -; RUN: llc --mtriple=aarch64 --mattr=+fullfp16 < %s | FileCheck %s --check-prefix=AARCH64 +; RUN: llc --mtriple=aarch64 --mattr=+fullfp16 < %s | FileCheck %s --check-prefixes=AARCH64,FULLFP16 +; RUN: llc --mtriple=aarch64 < %s | FileCheck %s --check-prefixes=AARCH64,NOFULLFP16 ;;;;;;;;;;;;;;;; max_f64 define double @max_nnan_f64(double %a, double %b) { @@ -142,96 +143,397 @@ entry: ;;;;;;;;;;;;;;;;;; max_f16 define half @max_nnan_f16(half %a, half %b) { -; AARCH64-LABEL: max_nnan_f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fmaxnm h0, h0, h1 -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_nnan_f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fmaxnm h0, h0, h1 +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_nnan_f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: ret entry: %c = call nnan half @llvm.maximumnum.f16(half %a, half %b) ret half %c } define <2 x half> @max_nnan_v2f16(<2 x half> %a, <2 x half> %b) { -; AARCH64-LABEL: max_nnan_v2f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fmaxnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_nnan_v2f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fmaxnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_nnan_v2f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fmaxnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fmaxnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fmaxnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fmaxnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call nnan <2 x half> @llvm.maximumnum.v2f16(<2 x half> %a, <2 x half> %b) ret <2 x half> %c } define <4 x half> @max_nnan_v4f16(<4 x half> %a, <4 x half> %b) { -; AARCH64-LABEL: max_nnan_v4f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fmaxnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_nnan_v4f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fmaxnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_nnan_v4f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fmaxnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fmaxnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fmaxnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fmaxnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call nnan <4 x half> @llvm.maximumnum.v4f16(<4 x half> %a, <4 x half> %b) ret <4 x half> %c } define <8 x half> @max_nnan_v8f16(<8 x half> %a, <8 x half> %b) { -; AARCH64-LABEL: max_nnan_v8f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fmaxnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_nnan_v8f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fmaxnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_nnan_v8f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h1 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h6, v1.h[2] +; NOFULLFP16-NEXT: mov h7, v0.h[2] +; NOFULLFP16-NEXT: mov h16, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fmaxnm s4, s5, s4 +; NOFULLFP16-NEXT: mov h5, v0.h[3] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fmaxnm s3, s3, s2 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fcvt h2, s4 +; NOFULLFP16-NEXT: fmaxnm s4, s7, s6 +; NOFULLFP16-NEXT: mov h6, v1.h[4] +; NOFULLFP16-NEXT: mov h7, v0.h[4] +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fmaxnm s5, s5, s16 +; NOFULLFP16-NEXT: mov h16, v0.h[5] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v2.h[1], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: fcvt s6, h7 +; NOFULLFP16-NEXT: mov h7, v1.h[5] +; NOFULLFP16-NEXT: fcvt h5, s5 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: mov v2.h[2], v4.h[0] +; NOFULLFP16-NEXT: mov h4, v1.h[6] +; NOFULLFP16-NEXT: fmaxnm s3, s6, s3 +; NOFULLFP16-NEXT: mov h6, v0.h[6] +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov v2.h[3], v5.h[0] +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt s5, h6 +; NOFULLFP16-NEXT: fmaxnm s6, s16, s7 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: mov v2.h[4], v3.h[0] +; NOFULLFP16-NEXT: fmaxnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt h3, s6 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s1 +; NOFULLFP16-NEXT: mov v2.h[5], v3.h[0] +; NOFULLFP16-NEXT: fcvt h3, s4 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v2.h[6], v3.h[0] +; NOFULLFP16-NEXT: mov v2.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v2.16b +; NOFULLFP16-NEXT: ret entry: %c = call nnan <8 x half> @llvm.maximumnum.v8f16(<8 x half> %a, <8 x half> %b) ret <8 x half> %c } define <9 x half> @max_nnan_v9f16(<9 x half> %a, <9 x half> %b) { -; AARCH64-LABEL: max_nnan_v9f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: // kill: def $h0 killed $h0 def $q0 -; AARCH64-NEXT: // kill: def $h1 killed $h1 def $q1 -; AARCH64-NEXT: // kill: def $h2 killed $h2 def $q2 -; AARCH64-NEXT: add x9, sp, #16 -; AARCH64-NEXT: // kill: def $h3 killed $h3 def $q3 -; AARCH64-NEXT: // kill: def $h4 killed $h4 def $q4 -; AARCH64-NEXT: // kill: def $h5 killed $h5 def $q5 -; AARCH64-NEXT: // kill: def $h6 killed $h6 def $q6 -; AARCH64-NEXT: // kill: def $h7 killed $h7 def $q7 -; AARCH64-NEXT: mov v0.h[1], v1.h[0] -; AARCH64-NEXT: ldr h1, [sp, #8] -; AARCH64-NEXT: ld1 { v1.h }[1], [x9] -; AARCH64-NEXT: add x9, sp, #24 -; AARCH64-NEXT: mov v0.h[2], v2.h[0] -; AARCH64-NEXT: ldr h2, [sp, #72] -; AARCH64-NEXT: ld1 { v1.h }[2], [x9] -; AARCH64-NEXT: add x9, sp, #32 -; AARCH64-NEXT: mov v0.h[3], v3.h[0] -; AARCH64-NEXT: ld1 { v1.h }[3], [x9] -; AARCH64-NEXT: add x9, sp, #40 -; AARCH64-NEXT: ldr h3, [sp] -; AARCH64-NEXT: ld1 { v1.h }[4], [x9] -; AARCH64-NEXT: add x9, sp, #48 -; AARCH64-NEXT: fmaxnm v2.8h, v3.8h, v2.8h -; AARCH64-NEXT: mov v0.h[4], v4.h[0] -; AARCH64-NEXT: ld1 { v1.h }[5], [x9] -; AARCH64-NEXT: add x9, sp, #56 -; AARCH64-NEXT: str h2, [x8, #16] -; AARCH64-NEXT: mov v0.h[5], v5.h[0] -; AARCH64-NEXT: ld1 { v1.h }[6], [x9] -; AARCH64-NEXT: add x9, sp, #64 -; AARCH64-NEXT: mov v0.h[6], v6.h[0] -; AARCH64-NEXT: ld1 { v1.h }[7], [x9] -; AARCH64-NEXT: mov v0.h[7], v7.h[0] -; AARCH64-NEXT: fmaxnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: str q0, [x8] -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_nnan_v9f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: // kill: def $h0 killed $h0 def $q0 +; FULLFP16-NEXT: // kill: def $h1 killed $h1 def $q1 +; FULLFP16-NEXT: // kill: def $h2 killed $h2 def $q2 +; FULLFP16-NEXT: add x9, sp, #16 +; FULLFP16-NEXT: // kill: def $h3 killed $h3 def $q3 +; FULLFP16-NEXT: // kill: def $h4 killed $h4 def $q4 +; FULLFP16-NEXT: // kill: def $h5 killed $h5 def $q5 +; FULLFP16-NEXT: // kill: def $h6 killed $h6 def $q6 +; FULLFP16-NEXT: // kill: def $h7 killed $h7 def $q7 +; FULLFP16-NEXT: mov v0.h[1], v1.h[0] +; FULLFP16-NEXT: ldr h1, [sp, #8] +; FULLFP16-NEXT: ld1 { v1.h }[1], [x9] +; FULLFP16-NEXT: add x9, sp, #24 +; FULLFP16-NEXT: mov v0.h[2], v2.h[0] +; FULLFP16-NEXT: ldr h2, [sp, #72] +; FULLFP16-NEXT: ld1 { v1.h }[2], [x9] +; FULLFP16-NEXT: add x9, sp, #32 +; FULLFP16-NEXT: mov v0.h[3], v3.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[3], [x9] +; FULLFP16-NEXT: add x9, sp, #40 +; FULLFP16-NEXT: ldr h3, [sp] +; FULLFP16-NEXT: ld1 { v1.h }[4], [x9] +; FULLFP16-NEXT: add x9, sp, #48 +; FULLFP16-NEXT: fmaxnm v2.8h, v3.8h, v2.8h +; FULLFP16-NEXT: mov v0.h[4], v4.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[5], [x9] +; FULLFP16-NEXT: add x9, sp, #56 +; FULLFP16-NEXT: str h2, [x8, #16] +; FULLFP16-NEXT: mov v0.h[5], v5.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[6], [x9] +; FULLFP16-NEXT: add x9, sp, #64 +; FULLFP16-NEXT: mov v0.h[6], v6.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[7], [x9] +; FULLFP16-NEXT: mov v0.h[7], v7.h[0] +; FULLFP16-NEXT: fmaxnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: str q0, [x8] +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_nnan_v9f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: ldr h16, [sp, #16] +; NOFULLFP16-NEXT: ldr h17, [sp, #8] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: ldr h18, [sp, #24] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fmaxnm s1, s1, s16 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s17 +; NOFULLFP16-NEXT: ldr h16, [sp, #32] +; NOFULLFP16-NEXT: fmaxnm s2, s2, s18 +; NOFULLFP16-NEXT: ldr h17, [sp, #40] +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fmaxnm s3, s3, s16 +; NOFULLFP16-NEXT: fmaxnm s4, s4, s17 +; NOFULLFP16-NEXT: mov v0.h[1], v1.h[0] +; NOFULLFP16-NEXT: ldr h1, [sp, #48] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: ldr h2, [sp, #56] +; NOFULLFP16-NEXT: fmaxnm s1, s5, s1 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: ldr h5, [sp, #64] +; NOFULLFP16-NEXT: mov v0.h[3], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: ldr h6, [sp, #72] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: mov v0.h[4], v4.h[0] +; NOFULLFP16-NEXT: fmaxnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h5 +; NOFULLFP16-NEXT: fcvt s4, h7 +; NOFULLFP16-NEXT: ldr h5, [sp] +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: mov v0.h[5], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fmaxnm s2, s4, s3 +; NOFULLFP16-NEXT: fmaxnm s3, s5, s6 +; NOFULLFP16-NEXT: mov v0.h[6], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: mov v0.h[7], v1.h[0] +; NOFULLFP16-NEXT: str h2, [x8, #16] +; NOFULLFP16-NEXT: str q0, [x8] +; NOFULLFP16-NEXT: ret entry: %c = call nnan <9 x half> @llvm.maximumnum.v9f16(<9 x half> %a, <9 x half> %b) ret <9 x half> %c } define <16 x half> @max_nnan_v16f16(<16 x half> %a, <16 x half> %b) { -; AARCH64-LABEL: max_nnan_v16f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fmaxnm v1.8h, v1.8h, v3.8h -; AARCH64-NEXT: fmaxnm v0.8h, v0.8h, v2.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_nnan_v16f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fmaxnm v1.8h, v1.8h, v3.8h +; FULLFP16-NEXT: fmaxnm v0.8h, v0.8h, v2.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_nnan_v16f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h6, v2.h[1] +; NOFULLFP16-NEXT: mov h7, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h2 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h16, v3.h[1] +; NOFULLFP16-NEXT: mov h17, v1.h[1] +; NOFULLFP16-NEXT: mov h18, v2.h[2] +; NOFULLFP16-NEXT: mov h19, v0.h[2] +; NOFULLFP16-NEXT: fcvt s20, h3 +; NOFULLFP16-NEXT: fcvt s21, h1 +; NOFULLFP16-NEXT: mov h22, v3.h[2] +; NOFULLFP16-NEXT: mov h23, v1.h[2] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h24, v0.h[6] +; NOFULLFP16-NEXT: fmaxnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt s5, h16 +; NOFULLFP16-NEXT: fcvt s16, h17 +; NOFULLFP16-NEXT: fcvt s17, h18 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: mov h19, v0.h[3] +; NOFULLFP16-NEXT: fmaxnm s20, s21, s20 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: mov h22, v3.h[3] +; NOFULLFP16-NEXT: fmaxnm s6, s7, s6 +; NOFULLFP16-NEXT: mov h7, v2.h[3] +; NOFULLFP16-NEXT: mov h25, v1.h[6] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: fmaxnm s5, s16, s5 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: mov h23, v1.h[3] +; NOFULLFP16-NEXT: fmaxnm s17, s18, s17 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt h19, s5 +; NOFULLFP16-NEXT: fcvt h5, s20 +; NOFULLFP16-NEXT: fmaxnm s16, s16, s21 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: mov h21, v2.h[4] +; NOFULLFP16-NEXT: mov h23, v1.h[4] +; NOFULLFP16-NEXT: mov v4.h[1], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h22 +; NOFULLFP16-NEXT: mov h22, v0.h[4] +; NOFULLFP16-NEXT: fmaxnm s7, s18, s7 +; NOFULLFP16-NEXT: mov h18, v3.h[4] +; NOFULLFP16-NEXT: mov v5.h[1], v19.h[0] +; NOFULLFP16-NEXT: fcvt h16, s16 +; NOFULLFP16-NEXT: fmaxnm s6, s20, s6 +; NOFULLFP16-NEXT: mov v4.h[2], v17.h[0] +; NOFULLFP16-NEXT: fcvt s17, h21 +; NOFULLFP16-NEXT: fcvt s19, h22 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: mov h21, v2.h[5] +; NOFULLFP16-NEXT: mov h22, v0.h[5] +; NOFULLFP16-NEXT: mov v5.h[2], v16.h[0] +; NOFULLFP16-NEXT: mov h16, v3.h[5] +; NOFULLFP16-NEXT: mov h23, v1.h[5] +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: fmaxnm s17, s19, s17 +; NOFULLFP16-NEXT: mov h19, v2.h[6] +; NOFULLFP16-NEXT: mov v4.h[3], v7.h[0] +; NOFULLFP16-NEXT: fmaxnm s18, s20, s18 +; NOFULLFP16-NEXT: mov h20, v3.h[6] +; NOFULLFP16-NEXT: fcvt s7, h21 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: fcvt s22, h24 +; NOFULLFP16-NEXT: mov h2, v2.h[7] +; NOFULLFP16-NEXT: mov v5.h[3], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h16 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: fcvt s19, h19 +; NOFULLFP16-NEXT: fcvt s23, h25 +; NOFULLFP16-NEXT: fcvt h18, s18 +; NOFULLFP16-NEXT: fcvt s20, h20 +; NOFULLFP16-NEXT: mov h3, v3.h[7] +; NOFULLFP16-NEXT: fmaxnm s7, s21, s7 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fmaxnm s6, s16, s6 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: mov v4.h[4], v17.h[0] +; NOFULLFP16-NEXT: fmaxnm s16, s22, s19 +; NOFULLFP16-NEXT: mov v5.h[4], v18.h[0] +; NOFULLFP16-NEXT: fmaxnm s17, s23, s20 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s2 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt h2, s16 +; NOFULLFP16-NEXT: fmaxnm s1, s1, s3 +; NOFULLFP16-NEXT: mov v4.h[5], v7.h[0] +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v5.h[5], v6.h[0] +; NOFULLFP16-NEXT: fcvt h6, s17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v4.h[6], v2.h[0] +; NOFULLFP16-NEXT: mov v5.h[6], v6.h[0] +; NOFULLFP16-NEXT: mov v4.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v5.h[7], v1.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v4.16b +; NOFULLFP16-NEXT: mov v1.16b, v5.16b +; NOFULLFP16-NEXT: ret entry: %c = call nnan <16 x half> @llvm.maximumnum.v16f16(<16 x half> %a, <16 x half> %b) ret <16 x half> %c @@ -378,96 +680,397 @@ entry: ;;;;;;;;;;;;;;;;;; min_f16 define half @min_nnan_f16(half %a, half %b) { -; AARCH64-LABEL: min_nnan_f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm h0, h0, h1 -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_nnan_f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm h0, h0, h1 +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_nnan_f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fminnm s0, s0, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: ret entry: %c = call nnan half @llvm.minimumnum.f16(half %a, half %b) ret half %c } define <2 x half> @min_nnan_v2f16(<2 x half> %a, <2 x half> %b) { -; AARCH64-LABEL: min_nnan_v2f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_nnan_v2f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_nnan_v2f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fminnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fminnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fminnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fminnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call nnan <2 x half> @llvm.minimumnum.v2f16(<2 x half> %a, <2 x half> %b) ret <2 x half> %c } define <4 x half> @min_nnan_v4f16(<4 x half> %a, <4 x half> %b) { -; AARCH64-LABEL: min_nnan_v4f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_nnan_v4f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_nnan_v4f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fminnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fminnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fminnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fminnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call nnan <4 x half> @llvm.minimumnum.v4f16(<4 x half> %a, <4 x half> %b) ret <4 x half> %c } define <8 x half> @min_nnan_v8f16(<8 x half> %a, <8 x half> %b) { -; AARCH64-LABEL: min_nnan_v8f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_nnan_v8f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_nnan_v8f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h1 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h6, v1.h[2] +; NOFULLFP16-NEXT: mov h7, v0.h[2] +; NOFULLFP16-NEXT: mov h16, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fminnm s4, s5, s4 +; NOFULLFP16-NEXT: mov h5, v0.h[3] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fminnm s3, s3, s2 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fcvt h2, s4 +; NOFULLFP16-NEXT: fminnm s4, s7, s6 +; NOFULLFP16-NEXT: mov h6, v1.h[4] +; NOFULLFP16-NEXT: mov h7, v0.h[4] +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fminnm s5, s5, s16 +; NOFULLFP16-NEXT: mov h16, v0.h[5] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v2.h[1], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: fcvt s6, h7 +; NOFULLFP16-NEXT: mov h7, v1.h[5] +; NOFULLFP16-NEXT: fcvt h5, s5 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: mov v2.h[2], v4.h[0] +; NOFULLFP16-NEXT: mov h4, v1.h[6] +; NOFULLFP16-NEXT: fminnm s3, s6, s3 +; NOFULLFP16-NEXT: mov h6, v0.h[6] +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov v2.h[3], v5.h[0] +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt s5, h6 +; NOFULLFP16-NEXT: fminnm s6, s16, s7 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: mov v2.h[4], v3.h[0] +; NOFULLFP16-NEXT: fminnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt h3, s6 +; NOFULLFP16-NEXT: fminnm s0, s0, s1 +; NOFULLFP16-NEXT: mov v2.h[5], v3.h[0] +; NOFULLFP16-NEXT: fcvt h3, s4 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v2.h[6], v3.h[0] +; NOFULLFP16-NEXT: mov v2.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v2.16b +; NOFULLFP16-NEXT: ret entry: %c = call nnan <8 x half> @llvm.minimumnum.v8f16(<8 x half> %a, <8 x half> %b) ret <8 x half> %c } define <9 x half> @min_nnan_v9f16(<9 x half> %a, <9 x half> %b) { -; AARCH64-LABEL: min_nnan_v9f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: // kill: def $h0 killed $h0 def $q0 -; AARCH64-NEXT: // kill: def $h1 killed $h1 def $q1 -; AARCH64-NEXT: // kill: def $h2 killed $h2 def $q2 -; AARCH64-NEXT: add x9, sp, #16 -; AARCH64-NEXT: // kill: def $h3 killed $h3 def $q3 -; AARCH64-NEXT: // kill: def $h4 killed $h4 def $q4 -; AARCH64-NEXT: // kill: def $h5 killed $h5 def $q5 -; AARCH64-NEXT: // kill: def $h6 killed $h6 def $q6 -; AARCH64-NEXT: // kill: def $h7 killed $h7 def $q7 -; AARCH64-NEXT: mov v0.h[1], v1.h[0] -; AARCH64-NEXT: ldr h1, [sp, #8] -; AARCH64-NEXT: ld1 { v1.h }[1], [x9] -; AARCH64-NEXT: add x9, sp, #24 -; AARCH64-NEXT: mov v0.h[2], v2.h[0] -; AARCH64-NEXT: ldr h2, [sp, #72] -; AARCH64-NEXT: ld1 { v1.h }[2], [x9] -; AARCH64-NEXT: add x9, sp, #32 -; AARCH64-NEXT: mov v0.h[3], v3.h[0] -; AARCH64-NEXT: ld1 { v1.h }[3], [x9] -; AARCH64-NEXT: add x9, sp, #40 -; AARCH64-NEXT: ldr h3, [sp] -; AARCH64-NEXT: ld1 { v1.h }[4], [x9] -; AARCH64-NEXT: add x9, sp, #48 -; AARCH64-NEXT: fminnm v2.8h, v3.8h, v2.8h -; AARCH64-NEXT: mov v0.h[4], v4.h[0] -; AARCH64-NEXT: ld1 { v1.h }[5], [x9] -; AARCH64-NEXT: add x9, sp, #56 -; AARCH64-NEXT: str h2, [x8, #16] -; AARCH64-NEXT: mov v0.h[5], v5.h[0] -; AARCH64-NEXT: ld1 { v1.h }[6], [x9] -; AARCH64-NEXT: add x9, sp, #64 -; AARCH64-NEXT: mov v0.h[6], v6.h[0] -; AARCH64-NEXT: ld1 { v1.h }[7], [x9] -; AARCH64-NEXT: mov v0.h[7], v7.h[0] -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: str q0, [x8] -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_nnan_v9f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: // kill: def $h0 killed $h0 def $q0 +; FULLFP16-NEXT: // kill: def $h1 killed $h1 def $q1 +; FULLFP16-NEXT: // kill: def $h2 killed $h2 def $q2 +; FULLFP16-NEXT: add x9, sp, #16 +; FULLFP16-NEXT: // kill: def $h3 killed $h3 def $q3 +; FULLFP16-NEXT: // kill: def $h4 killed $h4 def $q4 +; FULLFP16-NEXT: // kill: def $h5 killed $h5 def $q5 +; FULLFP16-NEXT: // kill: def $h6 killed $h6 def $q6 +; FULLFP16-NEXT: // kill: def $h7 killed $h7 def $q7 +; FULLFP16-NEXT: mov v0.h[1], v1.h[0] +; FULLFP16-NEXT: ldr h1, [sp, #8] +; FULLFP16-NEXT: ld1 { v1.h }[1], [x9] +; FULLFP16-NEXT: add x9, sp, #24 +; FULLFP16-NEXT: mov v0.h[2], v2.h[0] +; FULLFP16-NEXT: ldr h2, [sp, #72] +; FULLFP16-NEXT: ld1 { v1.h }[2], [x9] +; FULLFP16-NEXT: add x9, sp, #32 +; FULLFP16-NEXT: mov v0.h[3], v3.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[3], [x9] +; FULLFP16-NEXT: add x9, sp, #40 +; FULLFP16-NEXT: ldr h3, [sp] +; FULLFP16-NEXT: ld1 { v1.h }[4], [x9] +; FULLFP16-NEXT: add x9, sp, #48 +; FULLFP16-NEXT: fminnm v2.8h, v3.8h, v2.8h +; FULLFP16-NEXT: mov v0.h[4], v4.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[5], [x9] +; FULLFP16-NEXT: add x9, sp, #56 +; FULLFP16-NEXT: str h2, [x8, #16] +; FULLFP16-NEXT: mov v0.h[5], v5.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[6], [x9] +; FULLFP16-NEXT: add x9, sp, #64 +; FULLFP16-NEXT: mov v0.h[6], v6.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[7], [x9] +; FULLFP16-NEXT: mov v0.h[7], v7.h[0] +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: str q0, [x8] +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_nnan_v9f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: ldr h16, [sp, #16] +; NOFULLFP16-NEXT: ldr h17, [sp, #8] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: ldr h18, [sp, #24] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fminnm s1, s1, s16 +; NOFULLFP16-NEXT: fminnm s0, s0, s17 +; NOFULLFP16-NEXT: ldr h16, [sp, #32] +; NOFULLFP16-NEXT: fminnm s2, s2, s18 +; NOFULLFP16-NEXT: ldr h17, [sp, #40] +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fminnm s3, s3, s16 +; NOFULLFP16-NEXT: fminnm s4, s4, s17 +; NOFULLFP16-NEXT: mov v0.h[1], v1.h[0] +; NOFULLFP16-NEXT: ldr h1, [sp, #48] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: ldr h2, [sp, #56] +; NOFULLFP16-NEXT: fminnm s1, s5, s1 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: ldr h5, [sp, #64] +; NOFULLFP16-NEXT: mov v0.h[3], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: ldr h6, [sp, #72] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: mov v0.h[4], v4.h[0] +; NOFULLFP16-NEXT: fminnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h5 +; NOFULLFP16-NEXT: fcvt s4, h7 +; NOFULLFP16-NEXT: ldr h5, [sp] +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: mov v0.h[5], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fminnm s2, s4, s3 +; NOFULLFP16-NEXT: fminnm s3, s5, s6 +; NOFULLFP16-NEXT: mov v0.h[6], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: mov v0.h[7], v1.h[0] +; NOFULLFP16-NEXT: str h2, [x8, #16] +; NOFULLFP16-NEXT: str q0, [x8] +; NOFULLFP16-NEXT: ret entry: %c = call nnan <9 x half> @llvm.minimumnum.v9f16(<9 x half> %a, <9 x half> %b) ret <9 x half> %c } define <16 x half> @min_nnan_v16f16(<16 x half> %a, <16 x half> %b) { -; AARCH64-LABEL: min_nnan_v16f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v3.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v2.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_nnan_v16f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v3.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v2.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_nnan_v16f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h6, v2.h[1] +; NOFULLFP16-NEXT: mov h7, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h2 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h16, v3.h[1] +; NOFULLFP16-NEXT: mov h17, v1.h[1] +; NOFULLFP16-NEXT: mov h18, v2.h[2] +; NOFULLFP16-NEXT: mov h19, v0.h[2] +; NOFULLFP16-NEXT: fcvt s20, h3 +; NOFULLFP16-NEXT: fcvt s21, h1 +; NOFULLFP16-NEXT: mov h22, v3.h[2] +; NOFULLFP16-NEXT: mov h23, v1.h[2] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h24, v0.h[6] +; NOFULLFP16-NEXT: fminnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt s5, h16 +; NOFULLFP16-NEXT: fcvt s16, h17 +; NOFULLFP16-NEXT: fcvt s17, h18 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: mov h19, v0.h[3] +; NOFULLFP16-NEXT: fminnm s20, s21, s20 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: mov h22, v3.h[3] +; NOFULLFP16-NEXT: fminnm s6, s7, s6 +; NOFULLFP16-NEXT: mov h7, v2.h[3] +; NOFULLFP16-NEXT: mov h25, v1.h[6] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: fminnm s5, s16, s5 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: mov h23, v1.h[3] +; NOFULLFP16-NEXT: fminnm s17, s18, s17 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt h19, s5 +; NOFULLFP16-NEXT: fcvt h5, s20 +; NOFULLFP16-NEXT: fminnm s16, s16, s21 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: mov h21, v2.h[4] +; NOFULLFP16-NEXT: mov h23, v1.h[4] +; NOFULLFP16-NEXT: mov v4.h[1], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h22 +; NOFULLFP16-NEXT: mov h22, v0.h[4] +; NOFULLFP16-NEXT: fminnm s7, s18, s7 +; NOFULLFP16-NEXT: mov h18, v3.h[4] +; NOFULLFP16-NEXT: mov v5.h[1], v19.h[0] +; NOFULLFP16-NEXT: fcvt h16, s16 +; NOFULLFP16-NEXT: fminnm s6, s20, s6 +; NOFULLFP16-NEXT: mov v4.h[2], v17.h[0] +; NOFULLFP16-NEXT: fcvt s17, h21 +; NOFULLFP16-NEXT: fcvt s19, h22 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: mov h21, v2.h[5] +; NOFULLFP16-NEXT: mov h22, v0.h[5] +; NOFULLFP16-NEXT: mov v5.h[2], v16.h[0] +; NOFULLFP16-NEXT: mov h16, v3.h[5] +; NOFULLFP16-NEXT: mov h23, v1.h[5] +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: fminnm s17, s19, s17 +; NOFULLFP16-NEXT: mov h19, v2.h[6] +; NOFULLFP16-NEXT: mov v4.h[3], v7.h[0] +; NOFULLFP16-NEXT: fminnm s18, s20, s18 +; NOFULLFP16-NEXT: mov h20, v3.h[6] +; NOFULLFP16-NEXT: fcvt s7, h21 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: fcvt s22, h24 +; NOFULLFP16-NEXT: mov h2, v2.h[7] +; NOFULLFP16-NEXT: mov v5.h[3], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h16 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: fcvt s19, h19 +; NOFULLFP16-NEXT: fcvt s23, h25 +; NOFULLFP16-NEXT: fcvt h18, s18 +; NOFULLFP16-NEXT: fcvt s20, h20 +; NOFULLFP16-NEXT: mov h3, v3.h[7] +; NOFULLFP16-NEXT: fminnm s7, s21, s7 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fminnm s6, s16, s6 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: mov v4.h[4], v17.h[0] +; NOFULLFP16-NEXT: fminnm s16, s22, s19 +; NOFULLFP16-NEXT: mov v5.h[4], v18.h[0] +; NOFULLFP16-NEXT: fminnm s17, s23, s20 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fminnm s0, s0, s2 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt h2, s16 +; NOFULLFP16-NEXT: fminnm s1, s1, s3 +; NOFULLFP16-NEXT: mov v4.h[5], v7.h[0] +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v5.h[5], v6.h[0] +; NOFULLFP16-NEXT: fcvt h6, s17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v4.h[6], v2.h[0] +; NOFULLFP16-NEXT: mov v5.h[6], v6.h[0] +; NOFULLFP16-NEXT: mov v4.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v5.h[7], v1.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v4.16b +; NOFULLFP16-NEXT: mov v1.16b, v5.16b +; NOFULLFP16-NEXT: ret entry: %c = call nnan <16 x half> @llvm.minimumnum.v16f16(<16 x half> %a, <16 x half> %b) ret <16 x half> %c @@ -642,112 +1245,413 @@ entry: ;;;;;;;;;;;;;;;;;; max_f16 define half @max_f16(half %a, half %b) { -; AARCH64-LABEL: max_f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm h1, h1, h1 -; AARCH64-NEXT: fminnm h0, h0, h0 -; AARCH64-NEXT: fmaxnm h0, h0, h1 -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm h1, h1, h1 +; FULLFP16-NEXT: fminnm h0, h0, h0 +; FULLFP16-NEXT: fmaxnm h0, h0, h1 +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: ret entry: %c = call half @llvm.maximumnum.f16(half %a, half %b) ret half %c } define <2 x half> @max_v2f16(<2 x half> %a, <2 x half> %b) { -; AARCH64-LABEL: max_v2f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v1.4h, v1.4h, v1.4h -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v0.4h -; AARCH64-NEXT: fmaxnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_v2f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v1.4h, v1.4h, v1.4h +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v0.4h +; FULLFP16-NEXT: fmaxnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_v2f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fmaxnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fmaxnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fmaxnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fmaxnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call <2 x half> @llvm.maximumnum.v2f16(<2 x half> %a, <2 x half> %b) ret <2 x half> %c } define <4 x half> @max_v4f16(<4 x half> %a, <4 x half> %b) { -; AARCH64-LABEL: max_v4f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v1.4h, v1.4h, v1.4h -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v0.4h -; AARCH64-NEXT: fmaxnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_v4f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v1.4h, v1.4h, v1.4h +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v0.4h +; FULLFP16-NEXT: fmaxnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_v4f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fmaxnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fmaxnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fmaxnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fmaxnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call <4 x half> @llvm.maximumnum.v4f16(<4 x half> %a, <4 x half> %b) ret <4 x half> %c } define <8 x half> @max_v8f16(<8 x half> %a, <8 x half> %b) { -; AARCH64-LABEL: max_v8f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v1.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v0.8h -; AARCH64-NEXT: fmaxnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_v8f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v1.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v0.8h +; FULLFP16-NEXT: fmaxnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_v8f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h1 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h6, v1.h[2] +; NOFULLFP16-NEXT: mov h7, v0.h[2] +; NOFULLFP16-NEXT: mov h16, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fmaxnm s4, s5, s4 +; NOFULLFP16-NEXT: mov h5, v0.h[3] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fmaxnm s3, s3, s2 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fcvt h2, s4 +; NOFULLFP16-NEXT: fmaxnm s4, s7, s6 +; NOFULLFP16-NEXT: mov h6, v1.h[4] +; NOFULLFP16-NEXT: mov h7, v0.h[4] +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fmaxnm s5, s5, s16 +; NOFULLFP16-NEXT: mov h16, v0.h[5] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v2.h[1], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: fcvt s6, h7 +; NOFULLFP16-NEXT: mov h7, v1.h[5] +; NOFULLFP16-NEXT: fcvt h5, s5 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: mov v2.h[2], v4.h[0] +; NOFULLFP16-NEXT: mov h4, v1.h[6] +; NOFULLFP16-NEXT: fmaxnm s3, s6, s3 +; NOFULLFP16-NEXT: mov h6, v0.h[6] +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov v2.h[3], v5.h[0] +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt s5, h6 +; NOFULLFP16-NEXT: fmaxnm s6, s16, s7 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: mov v2.h[4], v3.h[0] +; NOFULLFP16-NEXT: fmaxnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt h3, s6 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s1 +; NOFULLFP16-NEXT: mov v2.h[5], v3.h[0] +; NOFULLFP16-NEXT: fcvt h3, s4 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v2.h[6], v3.h[0] +; NOFULLFP16-NEXT: mov v2.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v2.16b +; NOFULLFP16-NEXT: ret entry: %c = call <8 x half> @llvm.maximumnum.v8f16(<8 x half> %a, <8 x half> %b) ret <8 x half> %c } define <9 x half> @max_v9f16(<9 x half> %a, <9 x half> %b) { -; AARCH64-LABEL: max_v9f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: // kill: def $h0 killed $h0 def $q0 -; AARCH64-NEXT: // kill: def $h1 killed $h1 def $q1 -; AARCH64-NEXT: // kill: def $h2 killed $h2 def $q2 -; AARCH64-NEXT: add x9, sp, #16 -; AARCH64-NEXT: // kill: def $h3 killed $h3 def $q3 -; AARCH64-NEXT: // kill: def $h4 killed $h4 def $q4 -; AARCH64-NEXT: // kill: def $h5 killed $h5 def $q5 -; AARCH64-NEXT: // kill: def $h6 killed $h6 def $q6 -; AARCH64-NEXT: // kill: def $h7 killed $h7 def $q7 -; AARCH64-NEXT: mov v0.h[1], v1.h[0] -; AARCH64-NEXT: ldr h1, [sp, #8] -; AARCH64-NEXT: ld1 { v1.h }[1], [x9] -; AARCH64-NEXT: add x9, sp, #24 -; AARCH64-NEXT: mov v0.h[2], v2.h[0] -; AARCH64-NEXT: ldr h2, [sp] -; AARCH64-NEXT: ld1 { v1.h }[2], [x9] -; AARCH64-NEXT: add x9, sp, #32 -; AARCH64-NEXT: fminnm v2.8h, v2.8h, v2.8h -; AARCH64-NEXT: mov v0.h[3], v3.h[0] -; AARCH64-NEXT: ld1 { v1.h }[3], [x9] -; AARCH64-NEXT: add x9, sp, #40 -; AARCH64-NEXT: ldr h3, [sp, #72] -; AARCH64-NEXT: ld1 { v1.h }[4], [x9] -; AARCH64-NEXT: add x9, sp, #48 -; AARCH64-NEXT: fminnm v3.8h, v3.8h, v3.8h -; AARCH64-NEXT: mov v0.h[4], v4.h[0] -; AARCH64-NEXT: ld1 { v1.h }[5], [x9] -; AARCH64-NEXT: add x9, sp, #56 -; AARCH64-NEXT: fmaxnm v2.8h, v2.8h, v3.8h -; AARCH64-NEXT: mov v0.h[5], v5.h[0] -; AARCH64-NEXT: ld1 { v1.h }[6], [x9] -; AARCH64-NEXT: add x9, sp, #64 -; AARCH64-NEXT: str h2, [x8, #16] -; AARCH64-NEXT: mov v0.h[6], v6.h[0] -; AARCH64-NEXT: ld1 { v1.h }[7], [x9] -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v1.8h -; AARCH64-NEXT: mov v0.h[7], v7.h[0] -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v0.8h -; AARCH64-NEXT: fmaxnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: str q0, [x8] -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_v9f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: // kill: def $h0 killed $h0 def $q0 +; FULLFP16-NEXT: // kill: def $h1 killed $h1 def $q1 +; FULLFP16-NEXT: // kill: def $h2 killed $h2 def $q2 +; FULLFP16-NEXT: add x9, sp, #16 +; FULLFP16-NEXT: // kill: def $h3 killed $h3 def $q3 +; FULLFP16-NEXT: // kill: def $h4 killed $h4 def $q4 +; FULLFP16-NEXT: // kill: def $h5 killed $h5 def $q5 +; FULLFP16-NEXT: // kill: def $h6 killed $h6 def $q6 +; FULLFP16-NEXT: // kill: def $h7 killed $h7 def $q7 +; FULLFP16-NEXT: mov v0.h[1], v1.h[0] +; FULLFP16-NEXT: ldr h1, [sp, #8] +; FULLFP16-NEXT: ld1 { v1.h }[1], [x9] +; FULLFP16-NEXT: add x9, sp, #24 +; FULLFP16-NEXT: mov v0.h[2], v2.h[0] +; FULLFP16-NEXT: ldr h2, [sp] +; FULLFP16-NEXT: ld1 { v1.h }[2], [x9] +; FULLFP16-NEXT: add x9, sp, #32 +; FULLFP16-NEXT: fminnm v2.8h, v2.8h, v2.8h +; FULLFP16-NEXT: mov v0.h[3], v3.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[3], [x9] +; FULLFP16-NEXT: add x9, sp, #40 +; FULLFP16-NEXT: ldr h3, [sp, #72] +; FULLFP16-NEXT: ld1 { v1.h }[4], [x9] +; FULLFP16-NEXT: add x9, sp, #48 +; FULLFP16-NEXT: fminnm v3.8h, v3.8h, v3.8h +; FULLFP16-NEXT: mov v0.h[4], v4.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[5], [x9] +; FULLFP16-NEXT: add x9, sp, #56 +; FULLFP16-NEXT: fmaxnm v2.8h, v2.8h, v3.8h +; FULLFP16-NEXT: mov v0.h[5], v5.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[6], [x9] +; FULLFP16-NEXT: add x9, sp, #64 +; FULLFP16-NEXT: str h2, [x8, #16] +; FULLFP16-NEXT: mov v0.h[6], v6.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[7], [x9] +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v1.8h +; FULLFP16-NEXT: mov v0.h[7], v7.h[0] +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v0.8h +; FULLFP16-NEXT: fmaxnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: str q0, [x8] +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_v9f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: ldr h16, [sp, #16] +; NOFULLFP16-NEXT: ldr h17, [sp, #8] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: ldr h18, [sp, #24] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fmaxnm s1, s1, s16 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s17 +; NOFULLFP16-NEXT: ldr h16, [sp, #32] +; NOFULLFP16-NEXT: fmaxnm s2, s2, s18 +; NOFULLFP16-NEXT: ldr h17, [sp, #40] +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fmaxnm s3, s3, s16 +; NOFULLFP16-NEXT: fmaxnm s4, s4, s17 +; NOFULLFP16-NEXT: mov v0.h[1], v1.h[0] +; NOFULLFP16-NEXT: ldr h1, [sp, #48] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: ldr h2, [sp, #56] +; NOFULLFP16-NEXT: fmaxnm s1, s5, s1 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: ldr h5, [sp, #64] +; NOFULLFP16-NEXT: mov v0.h[3], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: ldr h6, [sp, #72] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: mov v0.h[4], v4.h[0] +; NOFULLFP16-NEXT: fmaxnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h5 +; NOFULLFP16-NEXT: fcvt s4, h7 +; NOFULLFP16-NEXT: ldr h5, [sp] +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: mov v0.h[5], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fmaxnm s2, s4, s3 +; NOFULLFP16-NEXT: fmaxnm s3, s5, s6 +; NOFULLFP16-NEXT: mov v0.h[6], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: mov v0.h[7], v1.h[0] +; NOFULLFP16-NEXT: str h2, [x8, #16] +; NOFULLFP16-NEXT: str q0, [x8] +; NOFULLFP16-NEXT: ret entry: %c = call <9 x half> @llvm.maximumnum.v9f16(<9 x half> %a, <9 x half> %b) ret <9 x half> %c } define <16 x half> @max_v16f16(<16 x half> %a, <16 x half> %b) { -; AARCH64-LABEL: max_v16f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v2.8h, v2.8h, v2.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v0.8h -; AARCH64-NEXT: fminnm v3.8h, v3.8h, v3.8h -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v1.8h -; AARCH64-NEXT: fmaxnm v0.8h, v0.8h, v2.8h -; AARCH64-NEXT: fmaxnm v1.8h, v1.8h, v3.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: max_v16f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v2.8h, v2.8h, v2.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v0.8h +; FULLFP16-NEXT: fminnm v3.8h, v3.8h, v3.8h +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v1.8h +; FULLFP16-NEXT: fmaxnm v0.8h, v0.8h, v2.8h +; FULLFP16-NEXT: fmaxnm v1.8h, v1.8h, v3.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: max_v16f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h6, v2.h[1] +; NOFULLFP16-NEXT: mov h7, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h2 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h16, v3.h[1] +; NOFULLFP16-NEXT: mov h17, v1.h[1] +; NOFULLFP16-NEXT: mov h18, v2.h[2] +; NOFULLFP16-NEXT: mov h19, v0.h[2] +; NOFULLFP16-NEXT: fcvt s20, h3 +; NOFULLFP16-NEXT: fcvt s21, h1 +; NOFULLFP16-NEXT: mov h22, v3.h[2] +; NOFULLFP16-NEXT: mov h23, v1.h[2] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h24, v0.h[6] +; NOFULLFP16-NEXT: fmaxnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt s5, h16 +; NOFULLFP16-NEXT: fcvt s16, h17 +; NOFULLFP16-NEXT: fcvt s17, h18 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: mov h19, v0.h[3] +; NOFULLFP16-NEXT: fmaxnm s20, s21, s20 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: mov h22, v3.h[3] +; NOFULLFP16-NEXT: fmaxnm s6, s7, s6 +; NOFULLFP16-NEXT: mov h7, v2.h[3] +; NOFULLFP16-NEXT: mov h25, v1.h[6] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: fmaxnm s5, s16, s5 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: mov h23, v1.h[3] +; NOFULLFP16-NEXT: fmaxnm s17, s18, s17 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt h19, s5 +; NOFULLFP16-NEXT: fcvt h5, s20 +; NOFULLFP16-NEXT: fmaxnm s16, s16, s21 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: mov h21, v2.h[4] +; NOFULLFP16-NEXT: mov h23, v1.h[4] +; NOFULLFP16-NEXT: mov v4.h[1], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h22 +; NOFULLFP16-NEXT: mov h22, v0.h[4] +; NOFULLFP16-NEXT: fmaxnm s7, s18, s7 +; NOFULLFP16-NEXT: mov h18, v3.h[4] +; NOFULLFP16-NEXT: mov v5.h[1], v19.h[0] +; NOFULLFP16-NEXT: fcvt h16, s16 +; NOFULLFP16-NEXT: fmaxnm s6, s20, s6 +; NOFULLFP16-NEXT: mov v4.h[2], v17.h[0] +; NOFULLFP16-NEXT: fcvt s17, h21 +; NOFULLFP16-NEXT: fcvt s19, h22 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: mov h21, v2.h[5] +; NOFULLFP16-NEXT: mov h22, v0.h[5] +; NOFULLFP16-NEXT: mov v5.h[2], v16.h[0] +; NOFULLFP16-NEXT: mov h16, v3.h[5] +; NOFULLFP16-NEXT: mov h23, v1.h[5] +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: fmaxnm s17, s19, s17 +; NOFULLFP16-NEXT: mov h19, v2.h[6] +; NOFULLFP16-NEXT: mov v4.h[3], v7.h[0] +; NOFULLFP16-NEXT: fmaxnm s18, s20, s18 +; NOFULLFP16-NEXT: mov h20, v3.h[6] +; NOFULLFP16-NEXT: fcvt s7, h21 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: fcvt s22, h24 +; NOFULLFP16-NEXT: mov h2, v2.h[7] +; NOFULLFP16-NEXT: mov v5.h[3], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h16 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: fcvt s19, h19 +; NOFULLFP16-NEXT: fcvt s23, h25 +; NOFULLFP16-NEXT: fcvt h18, s18 +; NOFULLFP16-NEXT: fcvt s20, h20 +; NOFULLFP16-NEXT: mov h3, v3.h[7] +; NOFULLFP16-NEXT: fmaxnm s7, s21, s7 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fmaxnm s6, s16, s6 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: mov v4.h[4], v17.h[0] +; NOFULLFP16-NEXT: fmaxnm s16, s22, s19 +; NOFULLFP16-NEXT: mov v5.h[4], v18.h[0] +; NOFULLFP16-NEXT: fmaxnm s17, s23, s20 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fmaxnm s0, s0, s2 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt h2, s16 +; NOFULLFP16-NEXT: fmaxnm s1, s1, s3 +; NOFULLFP16-NEXT: mov v4.h[5], v7.h[0] +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v5.h[5], v6.h[0] +; NOFULLFP16-NEXT: fcvt h6, s17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v4.h[6], v2.h[0] +; NOFULLFP16-NEXT: mov v5.h[6], v6.h[0] +; NOFULLFP16-NEXT: mov v4.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v5.h[7], v1.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v4.16b +; NOFULLFP16-NEXT: mov v1.16b, v5.16b +; NOFULLFP16-NEXT: ret entry: %c = call <16 x half> @llvm.maximumnum.v16f16(<16 x half> %a, <16 x half> %b) ret <16 x half> %c @@ -922,112 +1826,413 @@ entry: ;;;;;;;;;;;;;;;;;; min_f16 define half @min_f16(half %a, half %b) { -; AARCH64-LABEL: min_f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm h1, h1, h1 -; AARCH64-NEXT: fminnm h0, h0, h0 -; AARCH64-NEXT: fminnm h0, h0, h1 -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm h1, h1, h1 +; FULLFP16-NEXT: fminnm h0, h0, h0 +; FULLFP16-NEXT: fminnm h0, h0, h1 +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fminnm s0, s0, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: ret entry: %c = call half @llvm.minimumnum.f16(half %a, half %b) ret half %c } define <2 x half> @min_v2f16(<2 x half> %a, <2 x half> %b) { -; AARCH64-LABEL: min_v2f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v1.4h, v1.4h, v1.4h -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v0.4h -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_v2f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v1.4h, v1.4h, v1.4h +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v0.4h +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_v2f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fminnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fminnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fminnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fminnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call <2 x half> @llvm.minimumnum.v2f16(<2 x half> %a, <2 x half> %b) ret <2 x half> %c } define <4 x half> @min_v4f16(<4 x half> %a, <4 x half> %b) { -; AARCH64-LABEL: min_v4f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v1.4h, v1.4h, v1.4h -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v0.4h -; AARCH64-NEXT: fminnm v0.4h, v0.4h, v1.4h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_v4f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v1.4h, v1.4h, v1.4h +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v0.4h +; FULLFP16-NEXT: fminnm v0.4h, v0.4h, v1.4h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_v4f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: // kill: def $d1 killed $d1 def $q1 +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 def $q0 +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: mov h4, v1.h[2] +; NOFULLFP16-NEXT: mov h5, v0.h[2] +; NOFULLFP16-NEXT: fcvt s6, h1 +; NOFULLFP16-NEXT: fcvt s7, h0 +; NOFULLFP16-NEXT: mov h1, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fminnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h4 +; NOFULLFP16-NEXT: fcvt s4, h5 +; NOFULLFP16-NEXT: fminnm s5, s7, s6 +; NOFULLFP16-NEXT: mov h6, v0.h[3] +; NOFULLFP16-NEXT: fminnm s3, s4, s3 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fcvt h0, s5 +; NOFULLFP16-NEXT: fcvt s4, h6 +; NOFULLFP16-NEXT: mov v0.h[1], v2.h[0] +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: fminnm s1, s4, s1 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v0.h[3], v1.h[0] +; NOFULLFP16-NEXT: // kill: def $d0 killed $d0 killed $q0 +; NOFULLFP16-NEXT: ret entry: %c = call <4 x half> @llvm.minimumnum.v4f16(<4 x half> %a, <4 x half> %b) ret <4 x half> %c } define <8 x half> @min_v8f16(<8 x half> %a, <8 x half> %b) { -; AARCH64-LABEL: min_v8f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v1.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v0.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_v8f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v1.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v0.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_v8f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h2, v1.h[1] +; NOFULLFP16-NEXT: mov h3, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h1 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h6, v1.h[2] +; NOFULLFP16-NEXT: mov h7, v0.h[2] +; NOFULLFP16-NEXT: mov h16, v1.h[3] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fminnm s4, s5, s4 +; NOFULLFP16-NEXT: mov h5, v0.h[3] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fminnm s3, s3, s2 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fcvt h2, s4 +; NOFULLFP16-NEXT: fminnm s4, s7, s6 +; NOFULLFP16-NEXT: mov h6, v1.h[4] +; NOFULLFP16-NEXT: mov h7, v0.h[4] +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fminnm s5, s5, s16 +; NOFULLFP16-NEXT: mov h16, v0.h[5] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v2.h[1], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: fcvt s6, h7 +; NOFULLFP16-NEXT: mov h7, v1.h[5] +; NOFULLFP16-NEXT: fcvt h5, s5 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: mov v2.h[2], v4.h[0] +; NOFULLFP16-NEXT: mov h4, v1.h[6] +; NOFULLFP16-NEXT: fminnm s3, s6, s3 +; NOFULLFP16-NEXT: mov h6, v0.h[6] +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov v2.h[3], v5.h[0] +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt s5, h6 +; NOFULLFP16-NEXT: fminnm s6, s16, s7 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: mov v2.h[4], v3.h[0] +; NOFULLFP16-NEXT: fminnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt h3, s6 +; NOFULLFP16-NEXT: fminnm s0, s0, s1 +; NOFULLFP16-NEXT: mov v2.h[5], v3.h[0] +; NOFULLFP16-NEXT: fcvt h3, s4 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v2.h[6], v3.h[0] +; NOFULLFP16-NEXT: mov v2.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v2.16b +; NOFULLFP16-NEXT: ret entry: %c = call <8 x half> @llvm.minimumnum.v8f16(<8 x half> %a, <8 x half> %b) ret <8 x half> %c } define <9 x half> @min_v9f16(<9 x half> %a, <9 x half> %b) { -; AARCH64-LABEL: min_v9f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: // kill: def $h0 killed $h0 def $q0 -; AARCH64-NEXT: // kill: def $h1 killed $h1 def $q1 -; AARCH64-NEXT: // kill: def $h2 killed $h2 def $q2 -; AARCH64-NEXT: add x9, sp, #16 -; AARCH64-NEXT: // kill: def $h3 killed $h3 def $q3 -; AARCH64-NEXT: // kill: def $h4 killed $h4 def $q4 -; AARCH64-NEXT: // kill: def $h5 killed $h5 def $q5 -; AARCH64-NEXT: // kill: def $h6 killed $h6 def $q6 -; AARCH64-NEXT: // kill: def $h7 killed $h7 def $q7 -; AARCH64-NEXT: mov v0.h[1], v1.h[0] -; AARCH64-NEXT: ldr h1, [sp, #8] -; AARCH64-NEXT: ld1 { v1.h }[1], [x9] -; AARCH64-NEXT: add x9, sp, #24 -; AARCH64-NEXT: mov v0.h[2], v2.h[0] -; AARCH64-NEXT: ldr h2, [sp] -; AARCH64-NEXT: ld1 { v1.h }[2], [x9] -; AARCH64-NEXT: add x9, sp, #32 -; AARCH64-NEXT: fminnm v2.8h, v2.8h, v2.8h -; AARCH64-NEXT: mov v0.h[3], v3.h[0] -; AARCH64-NEXT: ld1 { v1.h }[3], [x9] -; AARCH64-NEXT: add x9, sp, #40 -; AARCH64-NEXT: ldr h3, [sp, #72] -; AARCH64-NEXT: ld1 { v1.h }[4], [x9] -; AARCH64-NEXT: add x9, sp, #48 -; AARCH64-NEXT: fminnm v3.8h, v3.8h, v3.8h -; AARCH64-NEXT: mov v0.h[4], v4.h[0] -; AARCH64-NEXT: ld1 { v1.h }[5], [x9] -; AARCH64-NEXT: add x9, sp, #56 -; AARCH64-NEXT: fminnm v2.8h, v2.8h, v3.8h -; AARCH64-NEXT: mov v0.h[5], v5.h[0] -; AARCH64-NEXT: ld1 { v1.h }[6], [x9] -; AARCH64-NEXT: add x9, sp, #64 -; AARCH64-NEXT: str h2, [x8, #16] -; AARCH64-NEXT: mov v0.h[6], v6.h[0] -; AARCH64-NEXT: ld1 { v1.h }[7], [x9] -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v1.8h -; AARCH64-NEXT: mov v0.h[7], v7.h[0] -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v0.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v1.8h -; AARCH64-NEXT: str q0, [x8] -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_v9f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: // kill: def $h0 killed $h0 def $q0 +; FULLFP16-NEXT: // kill: def $h1 killed $h1 def $q1 +; FULLFP16-NEXT: // kill: def $h2 killed $h2 def $q2 +; FULLFP16-NEXT: add x9, sp, #16 +; FULLFP16-NEXT: // kill: def $h3 killed $h3 def $q3 +; FULLFP16-NEXT: // kill: def $h4 killed $h4 def $q4 +; FULLFP16-NEXT: // kill: def $h5 killed $h5 def $q5 +; FULLFP16-NEXT: // kill: def $h6 killed $h6 def $q6 +; FULLFP16-NEXT: // kill: def $h7 killed $h7 def $q7 +; FULLFP16-NEXT: mov v0.h[1], v1.h[0] +; FULLFP16-NEXT: ldr h1, [sp, #8] +; FULLFP16-NEXT: ld1 { v1.h }[1], [x9] +; FULLFP16-NEXT: add x9, sp, #24 +; FULLFP16-NEXT: mov v0.h[2], v2.h[0] +; FULLFP16-NEXT: ldr h2, [sp] +; FULLFP16-NEXT: ld1 { v1.h }[2], [x9] +; FULLFP16-NEXT: add x9, sp, #32 +; FULLFP16-NEXT: fminnm v2.8h, v2.8h, v2.8h +; FULLFP16-NEXT: mov v0.h[3], v3.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[3], [x9] +; FULLFP16-NEXT: add x9, sp, #40 +; FULLFP16-NEXT: ldr h3, [sp, #72] +; FULLFP16-NEXT: ld1 { v1.h }[4], [x9] +; FULLFP16-NEXT: add x9, sp, #48 +; FULLFP16-NEXT: fminnm v3.8h, v3.8h, v3.8h +; FULLFP16-NEXT: mov v0.h[4], v4.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[5], [x9] +; FULLFP16-NEXT: add x9, sp, #56 +; FULLFP16-NEXT: fminnm v2.8h, v2.8h, v3.8h +; FULLFP16-NEXT: mov v0.h[5], v5.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[6], [x9] +; FULLFP16-NEXT: add x9, sp, #64 +; FULLFP16-NEXT: str h2, [x8, #16] +; FULLFP16-NEXT: mov v0.h[6], v6.h[0] +; FULLFP16-NEXT: ld1 { v1.h }[7], [x9] +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v1.8h +; FULLFP16-NEXT: mov v0.h[7], v7.h[0] +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v0.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v1.8h +; FULLFP16-NEXT: str q0, [x8] +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_v9f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: ldr h16, [sp, #16] +; NOFULLFP16-NEXT: ldr h17, [sp, #8] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: ldr h18, [sp, #24] +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s4, h4 +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: fminnm s1, s1, s16 +; NOFULLFP16-NEXT: fminnm s0, s0, s17 +; NOFULLFP16-NEXT: ldr h16, [sp, #32] +; NOFULLFP16-NEXT: fminnm s2, s2, s18 +; NOFULLFP16-NEXT: ldr h17, [sp, #40] +; NOFULLFP16-NEXT: fcvt s16, h16 +; NOFULLFP16-NEXT: fcvt s17, h17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: fcvt h2, s2 +; NOFULLFP16-NEXT: fminnm s3, s3, s16 +; NOFULLFP16-NEXT: fminnm s4, s4, s17 +; NOFULLFP16-NEXT: mov v0.h[1], v1.h[0] +; NOFULLFP16-NEXT: ldr h1, [sp, #48] +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: fcvt h3, s3 +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: mov v0.h[2], v2.h[0] +; NOFULLFP16-NEXT: ldr h2, [sp, #56] +; NOFULLFP16-NEXT: fminnm s1, s5, s1 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: ldr h5, [sp, #64] +; NOFULLFP16-NEXT: mov v0.h[3], v3.h[0] +; NOFULLFP16-NEXT: fcvt s3, h6 +; NOFULLFP16-NEXT: ldr h6, [sp, #72] +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: mov v0.h[4], v4.h[0] +; NOFULLFP16-NEXT: fminnm s2, s3, s2 +; NOFULLFP16-NEXT: fcvt s3, h5 +; NOFULLFP16-NEXT: fcvt s4, h7 +; NOFULLFP16-NEXT: ldr h5, [sp] +; NOFULLFP16-NEXT: fcvt s5, h5 +; NOFULLFP16-NEXT: mov v0.h[5], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fminnm s2, s4, s3 +; NOFULLFP16-NEXT: fminnm s3, s5, s6 +; NOFULLFP16-NEXT: mov v0.h[6], v1.h[0] +; NOFULLFP16-NEXT: fcvt h1, s2 +; NOFULLFP16-NEXT: fcvt h2, s3 +; NOFULLFP16-NEXT: mov v0.h[7], v1.h[0] +; NOFULLFP16-NEXT: str h2, [x8, #16] +; NOFULLFP16-NEXT: str q0, [x8] +; NOFULLFP16-NEXT: ret entry: %c = call <9 x half> @llvm.minimumnum.v9f16(<9 x half> %a, <9 x half> %b) ret <9 x half> %c } define <16 x half> @min_v16f16(<16 x half> %a, <16 x half> %b) { -; AARCH64-LABEL: min_v16f16: -; AARCH64: // %bb.0: // %entry -; AARCH64-NEXT: fminnm v2.8h, v2.8h, v2.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v0.8h -; AARCH64-NEXT: fminnm v3.8h, v3.8h, v3.8h -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v1.8h -; AARCH64-NEXT: fminnm v0.8h, v0.8h, v2.8h -; AARCH64-NEXT: fminnm v1.8h, v1.8h, v3.8h -; AARCH64-NEXT: ret +; FULLFP16-LABEL: min_v16f16: +; FULLFP16: // %bb.0: // %entry +; FULLFP16-NEXT: fminnm v2.8h, v2.8h, v2.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v0.8h +; FULLFP16-NEXT: fminnm v3.8h, v3.8h, v3.8h +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v1.8h +; FULLFP16-NEXT: fminnm v0.8h, v0.8h, v2.8h +; FULLFP16-NEXT: fminnm v1.8h, v1.8h, v3.8h +; FULLFP16-NEXT: ret +; +; NOFULLFP16-LABEL: min_v16f16: +; NOFULLFP16: // %bb.0: // %entry +; NOFULLFP16-NEXT: mov h6, v2.h[1] +; NOFULLFP16-NEXT: mov h7, v0.h[1] +; NOFULLFP16-NEXT: fcvt s4, h2 +; NOFULLFP16-NEXT: fcvt s5, h0 +; NOFULLFP16-NEXT: mov h16, v3.h[1] +; NOFULLFP16-NEXT: mov h17, v1.h[1] +; NOFULLFP16-NEXT: mov h18, v2.h[2] +; NOFULLFP16-NEXT: mov h19, v0.h[2] +; NOFULLFP16-NEXT: fcvt s20, h3 +; NOFULLFP16-NEXT: fcvt s21, h1 +; NOFULLFP16-NEXT: mov h22, v3.h[2] +; NOFULLFP16-NEXT: mov h23, v1.h[2] +; NOFULLFP16-NEXT: fcvt s6, h6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: mov h24, v0.h[6] +; NOFULLFP16-NEXT: fminnm s4, s5, s4 +; NOFULLFP16-NEXT: fcvt s5, h16 +; NOFULLFP16-NEXT: fcvt s16, h17 +; NOFULLFP16-NEXT: fcvt s17, h18 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: mov h19, v0.h[3] +; NOFULLFP16-NEXT: fminnm s20, s21, s20 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: mov h22, v3.h[3] +; NOFULLFP16-NEXT: fminnm s6, s7, s6 +; NOFULLFP16-NEXT: mov h7, v2.h[3] +; NOFULLFP16-NEXT: mov h25, v1.h[6] +; NOFULLFP16-NEXT: fcvt h4, s4 +; NOFULLFP16-NEXT: fminnm s5, s16, s5 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: mov h23, v1.h[3] +; NOFULLFP16-NEXT: fminnm s17, s18, s17 +; NOFULLFP16-NEXT: fcvt s18, h19 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt s7, h7 +; NOFULLFP16-NEXT: fcvt h19, s5 +; NOFULLFP16-NEXT: fcvt h5, s20 +; NOFULLFP16-NEXT: fminnm s16, s16, s21 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: mov h21, v2.h[4] +; NOFULLFP16-NEXT: mov h23, v1.h[4] +; NOFULLFP16-NEXT: mov v4.h[1], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h22 +; NOFULLFP16-NEXT: mov h22, v0.h[4] +; NOFULLFP16-NEXT: fminnm s7, s18, s7 +; NOFULLFP16-NEXT: mov h18, v3.h[4] +; NOFULLFP16-NEXT: mov v5.h[1], v19.h[0] +; NOFULLFP16-NEXT: fcvt h16, s16 +; NOFULLFP16-NEXT: fminnm s6, s20, s6 +; NOFULLFP16-NEXT: mov v4.h[2], v17.h[0] +; NOFULLFP16-NEXT: fcvt s17, h21 +; NOFULLFP16-NEXT: fcvt s19, h22 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fcvt s18, h18 +; NOFULLFP16-NEXT: fcvt s20, h23 +; NOFULLFP16-NEXT: mov h21, v2.h[5] +; NOFULLFP16-NEXT: mov h22, v0.h[5] +; NOFULLFP16-NEXT: mov v5.h[2], v16.h[0] +; NOFULLFP16-NEXT: mov h16, v3.h[5] +; NOFULLFP16-NEXT: mov h23, v1.h[5] +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: mov h0, v0.h[7] +; NOFULLFP16-NEXT: mov h1, v1.h[7] +; NOFULLFP16-NEXT: fminnm s17, s19, s17 +; NOFULLFP16-NEXT: mov h19, v2.h[6] +; NOFULLFP16-NEXT: mov v4.h[3], v7.h[0] +; NOFULLFP16-NEXT: fminnm s18, s20, s18 +; NOFULLFP16-NEXT: mov h20, v3.h[6] +; NOFULLFP16-NEXT: fcvt s7, h21 +; NOFULLFP16-NEXT: fcvt s21, h22 +; NOFULLFP16-NEXT: fcvt s22, h24 +; NOFULLFP16-NEXT: mov h2, v2.h[7] +; NOFULLFP16-NEXT: mov v5.h[3], v6.h[0] +; NOFULLFP16-NEXT: fcvt s6, h16 +; NOFULLFP16-NEXT: fcvt s16, h23 +; NOFULLFP16-NEXT: fcvt h17, s17 +; NOFULLFP16-NEXT: fcvt s19, h19 +; NOFULLFP16-NEXT: fcvt s23, h25 +; NOFULLFP16-NEXT: fcvt h18, s18 +; NOFULLFP16-NEXT: fcvt s20, h20 +; NOFULLFP16-NEXT: mov h3, v3.h[7] +; NOFULLFP16-NEXT: fminnm s7, s21, s7 +; NOFULLFP16-NEXT: fcvt s2, h2 +; NOFULLFP16-NEXT: fcvt s0, h0 +; NOFULLFP16-NEXT: fminnm s6, s16, s6 +; NOFULLFP16-NEXT: fcvt s1, h1 +; NOFULLFP16-NEXT: mov v4.h[4], v17.h[0] +; NOFULLFP16-NEXT: fminnm s16, s22, s19 +; NOFULLFP16-NEXT: mov v5.h[4], v18.h[0] +; NOFULLFP16-NEXT: fminnm s17, s23, s20 +; NOFULLFP16-NEXT: fcvt s3, h3 +; NOFULLFP16-NEXT: fcvt h7, s7 +; NOFULLFP16-NEXT: fminnm s0, s0, s2 +; NOFULLFP16-NEXT: fcvt h6, s6 +; NOFULLFP16-NEXT: fcvt h2, s16 +; NOFULLFP16-NEXT: fminnm s1, s1, s3 +; NOFULLFP16-NEXT: mov v4.h[5], v7.h[0] +; NOFULLFP16-NEXT: fcvt h0, s0 +; NOFULLFP16-NEXT: mov v5.h[5], v6.h[0] +; NOFULLFP16-NEXT: fcvt h6, s17 +; NOFULLFP16-NEXT: fcvt h1, s1 +; NOFULLFP16-NEXT: mov v4.h[6], v2.h[0] +; NOFULLFP16-NEXT: mov v5.h[6], v6.h[0] +; NOFULLFP16-NEXT: mov v4.h[7], v0.h[0] +; NOFULLFP16-NEXT: mov v5.h[7], v1.h[0] +; NOFULLFP16-NEXT: mov v0.16b, v4.16b +; NOFULLFP16-NEXT: mov v1.16b, v5.16b +; NOFULLFP16-NEXT: ret entry: %c = call <16 x half> @llvm.minimumnum.v16f16(<16 x half> %a, <16 x half> %b) ret <16 x half> %c From 4c97c5131f9ca32ce644a0be6e3586077ee03aa6 Mon Sep 17 00:00:00 2001 From: Sudharsan Veeravalli Date: Wed, 16 Apr 2025 06:48:56 +0530 Subject: [PATCH 50/63] [RISCV] Add ISel patterns for Xqcilia instructions (#135724) This patch adds instruction selection patterns for generating the long immediate arithmetic instructions. We prefer generating instructions that have a 26 bit immediate to a 32 bit immediate given that both are of the same size but the former might be easier to register allocate for. Base RISC-V arithmetic instructions will be preferred, when applicable. --- llvm/lib/Target/RISCV/RISCVInstrInfoXqci.td | 30 ++++++ llvm/test/CodeGen/RISCV/xqcilia.ll | 108 ++++++++++++++++++++ 2 files changed, 138 insertions(+) create mode 100644 llvm/test/CodeGen/RISCV/xqcilia.ll diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoXqci.td b/llvm/lib/Target/RISCV/RISCVInstrInfoXqci.td index 8eaa5e394a91c..2458bda80b1d6 100644 --- a/llvm/lib/Target/RISCV/RISCVInstrInfoXqci.td +++ b/llvm/lib/Target/RISCV/RISCVInstrInfoXqci.td @@ -118,6 +118,12 @@ def simm20_li : RISCVOp { def simm26 : RISCVSImmLeafOp<26>; +def simm26_nosimm12 : ImmLeaf(Imm) && !isInt<12>(Imm);}]>; + +def simm32_nosimm26 : ImmLeaf(Imm) && !isInt<26>(Imm);}]>; + class BareSImmNAsmOperand : ImmAsmOperand<"BareS", width, ""> { let PredicateMethod = "isBareSimmN<" # width # ">"; @@ -1223,5 +1229,29 @@ def PseudoQC_E_SW : PseudoStore<"qc.e.sw">; // Code Gen Patterns //===----------------------------------------------------------------------===// +/// Generic pattern classes + +class PatGprNoX0Simm26NoSimm12 + : Pat<(i32 (OpNode (i32 GPRNoX0:$rs1), simm26_nosimm12:$imm)), + (Inst GPRNoX0:$rs1, simm26_nosimm12:$imm)>; + +class PatGprNoX0Simm32NoSimm26 + : Pat<(i32 (OpNode (i32 GPRNoX0:$rs1), simm32_nosimm26:$imm)), + (Inst GPRNoX0:$rs1, simm32_nosimm26:$imm)>; + +/// Simple arithmetic operations + +let Predicates = [HasVendorXqcilia, IsRV32] in { +def : PatGprNoX0Simm32NoSimm26; +def : PatGprNoX0Simm32NoSimm26; +def : PatGprNoX0Simm32NoSimm26; +def : PatGprNoX0Simm32NoSimm26; + +def : PatGprNoX0Simm26NoSimm12; +def : PatGprNoX0Simm26NoSimm12; +def : PatGprNoX0Simm26NoSimm12; +def : PatGprNoX0Simm26NoSimm12; +} // Predicates = [HasVendorXqcilia, IsRV32] + let Predicates = [HasVendorXqciint, IsRV32] in def : Pat<(riscv_mileaveret_glue), (QC_C_MILEAVERET)>; diff --git a/llvm/test/CodeGen/RISCV/xqcilia.ll b/llvm/test/CodeGen/RISCV/xqcilia.ll new file mode 100644 index 0000000000000..0f14044d62dc8 --- /dev/null +++ b/llvm/test/CodeGen/RISCV/xqcilia.ll @@ -0,0 +1,108 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 +; Test that we are able to generate the Xqcilia instructions +; RUN: llc < %s -mtriple=riscv32 | FileCheck %s -check-prefix=RV32I +; RUN: llc < %s -mtriple=riscv32 -mattr=+experimental-xqcilia | FileCheck %s -check-prefix=RV32XQCILIA + +define i32 @add(i32 %a, i32 %b) { +; RV32I-LABEL: add: +; RV32I: # %bb.0: +; RV32I-NEXT: lui a2, 65536 +; RV32I-NEXT: add a0, a0, a2 +; RV32I-NEXT: lui a2, 573 +; RV32I-NEXT: addi a2, a2, -1330 +; RV32I-NEXT: add a1, a1, a2 +; RV32I-NEXT: and a0, a1, a0 +; RV32I-NEXT: addi a0, a0, 13 +; RV32I-NEXT: ret +; +; RV32XQCILIA-LABEL: add: +; RV32XQCILIA: # %bb.0: +; RV32XQCILIA-NEXT: qc.e.addi a1, a1, 2345678 +; RV32XQCILIA-NEXT: qc.e.addai a0, 268435456 +; RV32XQCILIA-NEXT: and a0, a0, a1 +; RV32XQCILIA-NEXT: addi a0, a0, 13 +; RV32XQCILIA-NEXT: ret + %addai = add i32 %a, 268435456 + %add = add i32 %b, 2345678 + %and = and i32 %add, %addai + %res = add i32 %and, 13 + ret i32 %res +} + +define i32 @and(i32 %a, i32 %b) { +; RV32I-LABEL: and: +; RV32I: # %bb.0: +; RV32I-NEXT: lui a2, 65536 +; RV32I-NEXT: and a0, a0, a2 +; RV32I-NEXT: lui a2, 573 +; RV32I-NEXT: addi a2, a2, -1330 +; RV32I-NEXT: and a1, a1, a2 +; RV32I-NEXT: srl a0, a1, a0 +; RV32I-NEXT: andi a0, a0, 10 +; RV32I-NEXT: ret +; +; RV32XQCILIA-LABEL: and: +; RV32XQCILIA: # %bb.0: +; RV32XQCILIA-NEXT: qc.e.andi a1, a1, 2345678 +; RV32XQCILIA-NEXT: qc.e.andai a0, 268435456 +; RV32XQCILIA-NEXT: srl a0, a1, a0 +; RV32XQCILIA-NEXT: andi a0, a0, 10 +; RV32XQCILIA-NEXT: ret + %andai = and i32 %a, 268435456 + %and = and i32 %b, 2345678 + %srl = lshr i32 %and, %andai + %res = and i32 %srl, 10 + ret i32 %res +} + +define i32 @or(i32 %a, i32 %b) { +; RV32I-LABEL: or: +; RV32I: # %bb.0: +; RV32I-NEXT: lui a2, 65536 +; RV32I-NEXT: or a0, a0, a2 +; RV32I-NEXT: lui a2, 573 +; RV32I-NEXT: addi a2, a2, -1330 +; RV32I-NEXT: or a1, a1, a2 +; RV32I-NEXT: add a0, a1, a0 +; RV32I-NEXT: ori a0, a0, 13 +; RV32I-NEXT: ret +; +; RV32XQCILIA-LABEL: or: +; RV32XQCILIA: # %bb.0: +; RV32XQCILIA-NEXT: qc.e.ori a1, a1, 2345678 +; RV32XQCILIA-NEXT: qc.e.orai a0, 268435456 +; RV32XQCILIA-NEXT: add a0, a0, a1 +; RV32XQCILIA-NEXT: ori a0, a0, 13 +; RV32XQCILIA-NEXT: ret + %orai = or i32 %a, 268435456 + %or = or i32 %b, 2345678 + %add = add i32 %or, %orai + %res = or i32 %add, 13 + ret i32 %res +} + +define i32 @xor(i32 %a, i32 %b) { +; RV32I-LABEL: xor: +; RV32I: # %bb.0: +; RV32I-NEXT: lui a2, 65536 +; RV32I-NEXT: xor a0, a0, a2 +; RV32I-NEXT: lui a2, 573 +; RV32I-NEXT: addi a2, a2, -1330 +; RV32I-NEXT: xor a1, a1, a2 +; RV32I-NEXT: add a0, a1, a0 +; RV32I-NEXT: xori a0, a0, 13 +; RV32I-NEXT: ret +; +; RV32XQCILIA-LABEL: xor: +; RV32XQCILIA: # %bb.0: +; RV32XQCILIA-NEXT: qc.e.xori a1, a1, 2345678 +; RV32XQCILIA-NEXT: qc.e.xorai a0, 268435456 +; RV32XQCILIA-NEXT: add a0, a0, a1 +; RV32XQCILIA-NEXT: xori a0, a0, 13 +; RV32XQCILIA-NEXT: ret + %xorai = xor i32 %a, 268435456 + %xor = xor i32 %b, 2345678 + %add = add i32 %xor, %xorai + %res = xor i32 %add, 13 + ret i32 %res +} From 0ce8ad68e44aaf50d1e2aa304fa8a1127e311e1d Mon Sep 17 00:00:00 2001 From: Koakuma Date: Wed, 16 Apr 2025 08:33:42 +0700 Subject: [PATCH 51/63] [SPARC] Use fzero/fzeros to materialize FP zeros when we have VIS Reviewers: rorth, brad0, s-barannikov Reviewed By: s-barannikov Pull Request: https://github.com/llvm/llvm-project/pull/135712 --- llvm/lib/Target/Sparc/SparcISelLowering.cpp | 6 + llvm/lib/Target/Sparc/SparcISelLowering.h | 3 + llvm/lib/Target/Sparc/SparcInstrVIS.td | 17 ++- llvm/test/CodeGen/SPARC/float-constants.ll | 115 ++++++++++++++++++++ 4 files changed, 139 insertions(+), 2 deletions(-) diff --git a/llvm/lib/Target/Sparc/SparcISelLowering.cpp b/llvm/lib/Target/Sparc/SparcISelLowering.cpp index 85b8750d40f46..bce8ddbd47586 100644 --- a/llvm/lib/Target/Sparc/SparcISelLowering.cpp +++ b/llvm/lib/Target/Sparc/SparcISelLowering.cpp @@ -3560,6 +3560,12 @@ bool SparcTargetLowering::useLoadStackGuardNode(const Module &M) const { return true; } +bool SparcTargetLowering::isFPImmLegal(const APFloat &Imm, EVT VT, + bool ForCodeSize) const { + return Subtarget->isVIS() && (VT == MVT::f32 || VT == MVT::f64) && + Imm.isZero(); +} + // Override to disable global variable loading on Linux. void SparcTargetLowering::insertSSPDeclarations(Module &M) const { if (!Subtarget->isTargetLinux()) diff --git a/llvm/lib/Target/Sparc/SparcISelLowering.h b/llvm/lib/Target/Sparc/SparcISelLowering.h index 1bee5f4cfe84d..c09e465f5d05e 100644 --- a/llvm/lib/Target/Sparc/SparcISelLowering.h +++ b/llvm/lib/Target/Sparc/SparcISelLowering.h @@ -207,6 +207,9 @@ namespace llvm { return VT != MVT::f128; } + bool isFPImmLegal(const APFloat &Imm, EVT VT, + bool ForCodeSize) const override; + bool shouldInsertFencesForAtomic(const Instruction *I) const override { // FIXME: We insert fences for each atomics and generate // sub-optimal code for PSO/TSO. (Approximately nobody uses any diff --git a/llvm/lib/Target/Sparc/SparcInstrVIS.td b/llvm/lib/Target/Sparc/SparcInstrVIS.td index 8ce8f37f34040..7be45fe9faf3f 100644 --- a/llvm/lib/Target/Sparc/SparcInstrVIS.td +++ b/llvm/lib/Target/Sparc/SparcInstrVIS.td @@ -45,10 +45,10 @@ class VISInst2 opfval, string OpcStr, RegisterClass RC = DFPRegs> !strconcat(OpcStr, " $rs2, $rd")>; // For VIS Instructions with only rd operand. -let Constraints = "$rd = $f", rs1 = 0, rs2 = 0 in +let rs1 = 0, rs2 = 0 in class VISInstD opfval, string OpcStr, RegisterClass RC = DFPRegs> : VISInstFormat; // VIS 1 Instructions @@ -277,3 +277,16 @@ def UMULXHI : VISInst<0b000010110, "umulxhi", I64Regs>; def XMULX : VISInst<0b100010101, "xmulx", I64Regs>; def XMULXHI : VISInst<0b100010110, "xmulxhi", I64Regs>; } // Predicates = [IsVIS3] + +// FP immediate patterns. +def fpimm0 : FPImmLeaf; +def fpnegimm0 : FPImmLeaf; + +// VIS instruction patterns. +let Predicates = [HasVIS] in { +// Zero immediate. +def : Pat<(f64 fpimm0), (FZERO)>; +def : Pat<(f32 fpimm0), (FZEROS)>; +def : Pat<(f64 fpnegimm0), (FNEGD (FZERO))>; +def : Pat<(f32 fpnegimm0), (FNEGS (FZEROS))>; +} // Predicates = [HasVIS] diff --git a/llvm/test/CodeGen/SPARC/float-constants.ll b/llvm/test/CodeGen/SPARC/float-constants.ll index b04ec68ed3d7e..440c75bfca9f9 100644 --- a/llvm/test/CodeGen/SPARC/float-constants.ll +++ b/llvm/test/CodeGen/SPARC/float-constants.ll @@ -1,6 +1,7 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2 ; RUN: llc < %s -mtriple=sparc | FileCheck %s ; RUN: llc < %s -mtriple=sparcel | FileCheck %s --check-prefix=CHECK-LE +; RUN: llc < %s -mtriple=sparcv9 -mattr=+vis | FileCheck %s --check-prefix=CHECK-VIS ;; Bitcast should not do a runtime conversion, but rather emit a ;; constant into integer registers directly. @@ -17,6 +18,12 @@ define <2 x i32> @bitcast() nounwind { ; CHECK-LE-NEXT: sethi 1049856, %o1 ; CHECK-LE-NEXT: retl ; CHECK-LE-NEXT: mov %g0, %o0 +; +; CHECK-VIS-LABEL: bitcast: +; CHECK-VIS: ! %bb.0: +; CHECK-VIS-NEXT: sethi 1049856, %o0 +; CHECK-VIS-NEXT: retl +; CHECK-VIS-NEXT: mov %g0, %o1 %1 = bitcast double 5.0 to <2 x i32> ret <2 x i32> %1 } @@ -43,6 +50,17 @@ define void @test_call() nounwind { ; CHECK-LE-NEXT: mov %g0, %o0 ; CHECK-LE-NEXT: ret ; CHECK-LE-NEXT: restore +; +; CHECK-VIS-LABEL: test_call: +; CHECK-VIS: ! %bb.0: +; CHECK-VIS-NEXT: save %sp, -176, %sp +; CHECK-VIS-NEXT: sethi %h44(.LCPI1_0), %i0 +; CHECK-VIS-NEXT: add %i0, %m44(.LCPI1_0), %i0 +; CHECK-VIS-NEXT: sllx %i0, 12, %i0 +; CHECK-VIS-NEXT: call a +; CHECK-VIS-NEXT: ldd [%i0+%l44(.LCPI1_0)], %f0 +; CHECK-VIS-NEXT: ret +; CHECK-VIS-NEXT: restore call void @a(double 5.0) ret void } @@ -75,6 +93,103 @@ define double @test_intrins_call() nounwind { ; CHECK-LE-NEXT: mov %o1, %o3 ; CHECK-LE-NEXT: ret ; CHECK-LE-NEXT: restore +; +; CHECK-VIS-LABEL: test_intrins_call: +; CHECK-VIS: ! %bb.0: +; CHECK-VIS-NEXT: save %sp, -176, %sp +; CHECK-VIS-NEXT: sethi %h44(.LCPI2_0), %i0 +; CHECK-VIS-NEXT: add %i0, %m44(.LCPI2_0), %i0 +; CHECK-VIS-NEXT: sllx %i0, 12, %i0 +; CHECK-VIS-NEXT: ldd [%i0+%l44(.LCPI2_0)], %f0 +; CHECK-VIS-NEXT: fmovd %f0, %f2 +; CHECK-VIS-NEXT: call pow +; CHECK-VIS-NEXT: nop +; CHECK-VIS-NEXT: ret +; CHECK-VIS-NEXT: restore %1 = call double @llvm.pow.f64(double 2.0, double 2.0) ret double %1 } + +;; When we have VIS, f32/f64 zero constant should be materialized from fzero/fzeros. + +define double @pos_zero_double() nounwind { +; CHECK-LABEL: pos_zero_double: +; CHECK: ! %bb.0: +; CHECK-NEXT: sethi %hi(.LCPI3_0), %o0 +; CHECK-NEXT: retl +; CHECK-NEXT: ldd [%o0+%lo(.LCPI3_0)], %f0 +; +; CHECK-LE-LABEL: pos_zero_double: +; CHECK-LE: ! %bb.0: +; CHECK-LE-NEXT: sethi %hi(.LCPI3_0), %o0 +; CHECK-LE-NEXT: retl +; CHECK-LE-NEXT: ldd [%o0+%lo(.LCPI3_0)], %f0 +; +; CHECK-VIS-LABEL: pos_zero_double: +; CHECK-VIS: ! %bb.0: +; CHECK-VIS-NEXT: retl +; CHECK-VIS-NEXT: fzero %f0 + ret double +0.0 +} + +define double @neg_zero_double() nounwind { +; CHECK-LABEL: neg_zero_double: +; CHECK: ! %bb.0: +; CHECK-NEXT: sethi %hi(.LCPI4_0), %o0 +; CHECK-NEXT: retl +; CHECK-NEXT: ldd [%o0+%lo(.LCPI4_0)], %f0 +; +; CHECK-LE-LABEL: neg_zero_double: +; CHECK-LE: ! %bb.0: +; CHECK-LE-NEXT: sethi %hi(.LCPI4_0), %o0 +; CHECK-LE-NEXT: retl +; CHECK-LE-NEXT: ldd [%o0+%lo(.LCPI4_0)], %f0 +; +; CHECK-VIS-LABEL: neg_zero_double: +; CHECK-VIS: ! %bb.0: +; CHECK-VIS-NEXT: fzero %f0 +; CHECK-VIS-NEXT: retl +; CHECK-VIS-NEXT: fnegd %f0, %f0 + ret double -0.0 +} + +define float @pos_zero_float() nounwind { +; CHECK-LABEL: pos_zero_float: +; CHECK: ! %bb.0: +; CHECK-NEXT: sethi %hi(.LCPI5_0), %o0 +; CHECK-NEXT: retl +; CHECK-NEXT: ld [%o0+%lo(.LCPI5_0)], %f0 +; +; CHECK-LE-LABEL: pos_zero_float: +; CHECK-LE: ! %bb.0: +; CHECK-LE-NEXT: sethi %hi(.LCPI5_0), %o0 +; CHECK-LE-NEXT: retl +; CHECK-LE-NEXT: ld [%o0+%lo(.LCPI5_0)], %f0 +; +; CHECK-VIS-LABEL: pos_zero_float: +; CHECK-VIS: ! %bb.0: +; CHECK-VIS-NEXT: retl +; CHECK-VIS-NEXT: fzeros %f0 + ret float +0.0 +} + +define float @neg_zero_float() nounwind { +; CHECK-LABEL: neg_zero_float: +; CHECK: ! %bb.0: +; CHECK-NEXT: sethi %hi(.LCPI6_0), %o0 +; CHECK-NEXT: retl +; CHECK-NEXT: ld [%o0+%lo(.LCPI6_0)], %f0 +; +; CHECK-LE-LABEL: neg_zero_float: +; CHECK-LE: ! %bb.0: +; CHECK-LE-NEXT: sethi %hi(.LCPI6_0), %o0 +; CHECK-LE-NEXT: retl +; CHECK-LE-NEXT: ld [%o0+%lo(.LCPI6_0)], %f0 +; +; CHECK-VIS-LABEL: neg_zero_float: +; CHECK-VIS: ! %bb.0: +; CHECK-VIS-NEXT: fzeros %f0 +; CHECK-VIS-NEXT: retl +; CHECK-VIS-NEXT: fnegs %f0, %f0 + ret float -0.0 +} From f3de63c64998eb46db9cf26aca9ebcc5453f6f44 Mon Sep 17 00:00:00 2001 From: Koakuma Date: Wed, 16 Apr 2025 08:36:47 +0700 Subject: [PATCH 52/63] [SPARC] Use addxccc to do multiword addition when we have VIS3 Reviewers: brad0, s-barannikov, rorth Reviewed By: s-barannikov Pull Request: https://github.com/llvm/llvm-project/pull/135713 --- llvm/lib/Target/Sparc/SparcISelLowering.cpp | 5 + llvm/lib/Target/Sparc/SparcInstr64Bit.td | 2 + llvm/lib/Target/Sparc/SparcInstrVIS.td | 5 + llvm/test/CodeGen/SPARC/2011-01-11-CC.ll | 118 ++++++++++++++++++++ 4 files changed, 130 insertions(+) diff --git a/llvm/lib/Target/Sparc/SparcISelLowering.cpp b/llvm/lib/Target/Sparc/SparcISelLowering.cpp index bce8ddbd47586..098e5f22834f4 100644 --- a/llvm/lib/Target/Sparc/SparcISelLowering.cpp +++ b/llvm/lib/Target/Sparc/SparcISelLowering.cpp @@ -1737,6 +1737,11 @@ SparcTargetLowering::SparcTargetLowering(const TargetMachine &TM, setOperationAction(ISD::SUBC, MVT::i32, Legal); setOperationAction(ISD::SUBE, MVT::i32, Legal); + if (Subtarget->isVIS3()) { + setOperationAction(ISD::ADDC, MVT::i64, Legal); + setOperationAction(ISD::ADDE, MVT::i64, Legal); + } + if (Subtarget->is64Bit()) { setOperationAction(ISD::BITCAST, MVT::f64, Expand); setOperationAction(ISD::BITCAST, MVT::i64, Expand); diff --git a/llvm/lib/Target/Sparc/SparcInstr64Bit.td b/llvm/lib/Target/Sparc/SparcInstr64Bit.td index 56fab2f26a19e..000612534e89d 100644 --- a/llvm/lib/Target/Sparc/SparcInstr64Bit.td +++ b/llvm/lib/Target/Sparc/SparcInstr64Bit.td @@ -157,9 +157,11 @@ def : Pat<(and i64:$lhs, (not i64:$rhs)), (ANDNrr $lhs, $rhs)>; def : Pat<(or i64:$lhs, (not i64:$rhs)), (ORNrr $lhs, $rhs)>; def : Pat<(not (xor i64:$lhs, i64:$rhs)), (XNORrr $lhs, $rhs)>; +def : Pat<(addc i64:$lhs, i64:$rhs), (ADDCCrr $lhs, $rhs)>, Requires<[HasVIS3]>; def : Pat<(add i64:$lhs, i64:$rhs), (ADDrr $lhs, $rhs)>; def : Pat<(sub i64:$lhs, i64:$rhs), (SUBrr $lhs, $rhs)>; +def : Pat<(addc i64:$lhs, (i64 simm13:$rhs)), (ADDCCri $lhs, imm:$rhs)>, Requires<[HasVIS3]>; def : Pat<(add i64:$lhs, (i64 simm13:$rhs)), (ADDri $lhs, imm:$rhs)>; def : Pat<(sub i64:$lhs, (i64 simm13:$rhs)), (SUBri $lhs, imm:$rhs)>; diff --git a/llvm/lib/Target/Sparc/SparcInstrVIS.td b/llvm/lib/Target/Sparc/SparcInstrVIS.td index 7be45fe9faf3f..ee24d8a54fe8e 100644 --- a/llvm/lib/Target/Sparc/SparcInstrVIS.td +++ b/llvm/lib/Target/Sparc/SparcInstrVIS.td @@ -290,3 +290,8 @@ def : Pat<(f32 fpimm0), (FZEROS)>; def : Pat<(f64 fpnegimm0), (FNEGD (FZERO))>; def : Pat<(f32 fpnegimm0), (FNEGS (FZEROS))>; } // Predicates = [HasVIS] + +// VIS3 instruction patterns. +let Predicates = [HasVIS3] in { +def : Pat<(i64 (adde i64:$lhs, i64:$rhs)), (ADDXCCC $lhs, $rhs)>; +} // Predicates = [HasVIS3] diff --git a/llvm/test/CodeGen/SPARC/2011-01-11-CC.ll b/llvm/test/CodeGen/SPARC/2011-01-11-CC.ll index 1560bc687b7dd..e05c47bfee766 100644 --- a/llvm/test/CodeGen/SPARC/2011-01-11-CC.ll +++ b/llvm/test/CodeGen/SPARC/2011-01-11-CC.ll @@ -2,6 +2,7 @@ ; RUN: llc -mtriple=sparc %s -o - | FileCheck %s -check-prefix=V8 ; RUN: llc -mtriple=sparc -mattr=v9 %s -o - | FileCheck %s -check-prefix=V9 ; RUN: llc -mtriple=sparc64-unknown-linux %s -o - | FileCheck %s -check-prefix=SPARC64 +; RUN: llc -mtriple=sparc64-unknown-linux -mattr=vis3 %s -o - | FileCheck %s -check-prefix=SPARC64-VIS3 define i32 @test_addx(i64 %a, i64 %b, i64 %c) nounwind { ; V8-LABEL: test_addx: @@ -60,6 +61,15 @@ define i32 @test_addx(i64 %a, i64 %b, i64 %c) nounwind { ; SPARC64-NEXT: movgu %xcc, 1, %o3 ; SPARC64-NEXT: retl ; SPARC64-NEXT: srl %o3, 0, %o0 +; +; SPARC64-VIS3-LABEL: test_addx: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: mov %g0, %o3 +; SPARC64-VIS3-NEXT: add %o0, %o1, %o0 +; SPARC64-VIS3-NEXT: cmp %o0, %o2 +; SPARC64-VIS3-NEXT: movgu %xcc, 1, %o3 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: srl %o3, 0, %o0 entry: %0 = add i64 %a, %b %1 = icmp ugt i64 %0, %c @@ -92,6 +102,13 @@ define i32 @test_select_int_icc(i32 %a, i32 %b, i32 %c) nounwind { ; SPARC64-NEXT: move %icc, %o1, %o2 ; SPARC64-NEXT: retl ; SPARC64-NEXT: mov %o2, %o0 +; +; SPARC64-VIS3-LABEL: test_select_int_icc: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: cmp %o0, 0 +; SPARC64-VIS3-NEXT: move %icc, %o1, %o2 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: mov %o2, %o0 entry: %0 = icmp eq i32 %a, 0 %1 = select i1 %0, i32 %b, i32 %c @@ -133,6 +150,13 @@ define float @test_select_fp_icc(i32 %a, float %f1, float %f2) nounwind { ; SPARC64-NEXT: cmp %o0, 0 ; SPARC64-NEXT: retl ; SPARC64-NEXT: fmovse %icc, %f3, %f0 +; +; SPARC64-VIS3-LABEL: test_select_fp_icc: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: fmovs %f5, %f0 +; SPARC64-VIS3-NEXT: cmp %o0, 0 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: fmovse %icc, %f3, %f0 entry: %0 = icmp eq i32 %a, 0 %1 = select i1 %0, float %f1, float %f2 @@ -182,6 +206,13 @@ define double @test_select_dfp_icc(i32 %a, double %f1, double %f2) nounwind { ; SPARC64-NEXT: cmp %o0, 0 ; SPARC64-NEXT: retl ; SPARC64-NEXT: fmovde %icc, %f2, %f0 +; +; SPARC64-VIS3-LABEL: test_select_dfp_icc: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: fmovd %f4, %f0 +; SPARC64-VIS3-NEXT: cmp %o0, 0 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: fmovde %icc, %f2, %f0 entry: %0 = icmp eq i32 %a, 0 %1 = select i1 %0, double %f1, double %f2 @@ -229,6 +260,17 @@ define i32 @test_select_int_fcc(float %f, i32 %a, i32 %b) nounwind { ; SPARC64-NEXT: fcmps %fcc0, %f1, %f0 ; SPARC64-NEXT: retl ; SPARC64-NEXT: movne %fcc0, %o1, %o0 +; +; SPARC64-VIS3-LABEL: test_select_int_fcc: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: sethi %h44(.LCPI4_0), %o0 +; SPARC64-VIS3-NEXT: add %o0, %m44(.LCPI4_0), %o0 +; SPARC64-VIS3-NEXT: sllx %o0, 12, %o0 +; SPARC64-VIS3-NEXT: ld [%o0+%l44(.LCPI4_0)], %f0 +; SPARC64-VIS3-NEXT: mov %o2, %o0 +; SPARC64-VIS3-NEXT: fcmps %fcc0, %f1, %f0 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: movne %fcc0, %o1, %o0 entry: %0 = fcmp une float %f, 0.000000e+00 %a.b = select i1 %0, i32 %a, i32 %b @@ -284,6 +326,17 @@ define float @test_select_fp_fcc(float %f, float %f1, float %f2) nounwind { ; SPARC64-NEXT: fcmps %fcc0, %f1, %f2 ; SPARC64-NEXT: retl ; SPARC64-NEXT: fmovsne %fcc0, %f3, %f0 +; +; SPARC64-VIS3-LABEL: test_select_fp_fcc: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: sethi %h44(.LCPI5_0), %o0 +; SPARC64-VIS3-NEXT: add %o0, %m44(.LCPI5_0), %o0 +; SPARC64-VIS3-NEXT: sllx %o0, 12, %o0 +; SPARC64-VIS3-NEXT: ld [%o0+%l44(.LCPI5_0)], %f2 +; SPARC64-VIS3-NEXT: fmovs %f5, %f0 +; SPARC64-VIS3-NEXT: fcmps %fcc0, %f1, %f2 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: fmovsne %fcc0, %f3, %f0 entry: %0 = fcmp une float %f, 0.000000e+00 %1 = select i1 %0, float %f1, float %f2 @@ -352,6 +405,18 @@ define double @test_select_dfp_fcc(double %f, double %f1, double %f2) nounwind { ; SPARC64-NEXT: fmovd %f4, %f0 ; SPARC64-NEXT: retl ; SPARC64-NEXT: nop +; +; SPARC64-VIS3-LABEL: test_select_dfp_fcc: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: sethi %h44(.LCPI6_0), %o0 +; SPARC64-VIS3-NEXT: add %o0, %m44(.LCPI6_0), %o0 +; SPARC64-VIS3-NEXT: sllx %o0, 12, %o0 +; SPARC64-VIS3-NEXT: ldd [%o0+%l44(.LCPI6_0)], %f6 +; SPARC64-VIS3-NEXT: fcmpd %fcc0, %f0, %f6 +; SPARC64-VIS3-NEXT: fmovdne %fcc0, %f2, %f4 +; SPARC64-VIS3-NEXT: fmovd %f4, %f0 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: nop entry: %0 = fcmp une double %f, 0.000000e+00 %1 = select i1 %0, double %f1, double %f2 @@ -453,6 +518,31 @@ define i32 @test_float_cc(double %a, double %b, i32 %c, i32 %d) nounwind { ; SPARC64-NEXT: ! %bb.4: ! %exit.0 ; SPARC64-NEXT: retl ; SPARC64-NEXT: mov %g0, %o0 +; +; SPARC64-VIS3-LABEL: test_float_cc: +; SPARC64-VIS3: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: sethi %h44(.LCPI7_0), %o0 +; SPARC64-VIS3-NEXT: add %o0, %m44(.LCPI7_0), %o0 +; SPARC64-VIS3-NEXT: sllx %o0, 12, %o0 +; SPARC64-VIS3-NEXT: ldd [%o0+%l44(.LCPI7_0)], %f4 +; SPARC64-VIS3-NEXT: fcmpd %fcc0, %f0, %f4 +; SPARC64-VIS3-NEXT: fbuge %fcc0, .LBB7_3 +; SPARC64-VIS3-NEXT: nop +; SPARC64-VIS3-NEXT: ! %bb.1: ! %loop.2 +; SPARC64-VIS3-NEXT: fcmpd %fcc0, %f2, %f4 +; SPARC64-VIS3-NEXT: fbule %fcc0, .LBB7_3 +; SPARC64-VIS3-NEXT: nop +; SPARC64-VIS3-NEXT: ! %bb.2: ! %exit.1 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: mov 1, %o0 +; SPARC64-VIS3-NEXT: .LBB7_3: ! %loop +; SPARC64-VIS3-NEXT: ! =>This Inner Loop Header: Depth=1 +; SPARC64-VIS3-NEXT: cmp %o2, 10 +; SPARC64-VIS3-NEXT: be %icc, .LBB7_3 +; SPARC64-VIS3-NEXT: nop +; SPARC64-VIS3-NEXT: ! %bb.4: ! %exit.0 +; SPARC64-VIS3-NEXT: retl +; SPARC64-VIS3-NEXT: mov %g0, %o0 entry: %0 = fcmp uge double %a, 0.000000e+00 br i1 %0, label %loop, label %loop.2 @@ -558,6 +648,34 @@ define void @test_adde_sube(ptr %a, ptr %b, ptr %sum, ptr %diff) nounwind { ; SPARC64-NEXT: stx %i0, [%i3] ; SPARC64-NEXT: ret ; SPARC64-NEXT: restore +; +; SPARC64-VIS3-LABEL: test_adde_sube: +; SPARC64-VIS3: .register %g2, #scratch +; SPARC64-VIS3-NEXT: ! %bb.0: ! %entry +; SPARC64-VIS3-NEXT: save %sp, -128, %sp +; SPARC64-VIS3-NEXT: ldx [%i0+8], %i4 +; SPARC64-VIS3-NEXT: ldx [%i0], %i5 +; SPARC64-VIS3-NEXT: ldx [%i1+8], %g2 +; SPARC64-VIS3-NEXT: ldx [%i1], %i1 +; SPARC64-VIS3-NEXT: addcc %i4, %g2, %g2 +; SPARC64-VIS3-NEXT: addxccc %i5, %i1, %i1 +; SPARC64-VIS3-NEXT: stx %i1, [%i2] +; SPARC64-VIS3-NEXT: stx %g2, [%i2+8] +; SPARC64-VIS3-NEXT: !APP +; SPARC64-VIS3-NEXT: !NO_APP +; SPARC64-VIS3-NEXT: ldx [%i0+8], %i1 +; SPARC64-VIS3-NEXT: mov %g0, %i2 +; SPARC64-VIS3-NEXT: ldx [%i0], %i0 +; SPARC64-VIS3-NEXT: cmp %i4, %i1 +; SPARC64-VIS3-NEXT: movcs %xcc, 1, %i2 +; SPARC64-VIS3-NEXT: srl %i2, 0, %i2 +; SPARC64-VIS3-NEXT: sub %i5, %i0, %i0 +; SPARC64-VIS3-NEXT: sub %i0, %i2, %i0 +; SPARC64-VIS3-NEXT: sub %i4, %i1, %i1 +; SPARC64-VIS3-NEXT: stx %i1, [%i3+8] +; SPARC64-VIS3-NEXT: stx %i0, [%i3] +; SPARC64-VIS3-NEXT: ret +; SPARC64-VIS3-NEXT: restore entry: %0 = bitcast ptr %a to ptr %1 = bitcast ptr %b to ptr From e4f2191f568db718ed67defa664f83f763e7e74a Mon Sep 17 00:00:00 2001 From: Koakuma Date: Wed, 16 Apr 2025 08:38:29 +0700 Subject: [PATCH 53/63] [SPARC] Use umulxhi to do extending 64x64->128 multiply when we have VIS3 Reviewers: s-barannikov, rorth, brad0 Reviewed By: s-barannikov Pull Request: https://github.com/llvm/llvm-project/pull/135714 --- llvm/lib/Target/Sparc/SparcISelLowering.cpp | 6 +- llvm/lib/Target/Sparc/SparcInstrVIS.td | 10 ++++ llvm/test/CodeGen/SPARC/multiply-extension.ll | 59 +++++++++++++++++++ .../SPARC/smulo-128-legalisation-lowering.ll | 44 ++++++++++++++ .../SPARC/umulo-128-legalisation-lowering.ll | 33 +++++++++++ 5 files changed, 150 insertions(+), 2 deletions(-) create mode 100644 llvm/test/CodeGen/SPARC/multiply-extension.ll diff --git a/llvm/lib/Target/Sparc/SparcISelLowering.cpp b/llvm/lib/Target/Sparc/SparcISelLowering.cpp index 098e5f22834f4..0ad261135651f 100644 --- a/llvm/lib/Target/Sparc/SparcISelLowering.cpp +++ b/llvm/lib/Target/Sparc/SparcISelLowering.cpp @@ -1854,8 +1854,10 @@ SparcTargetLowering::SparcTargetLowering(const TargetMachine &TM, if (Subtarget->is64Bit()) { setOperationAction(ISD::UMUL_LOHI, MVT::i64, Expand); setOperationAction(ISD::SMUL_LOHI, MVT::i64, Expand); - setOperationAction(ISD::MULHU, MVT::i64, Expand); - setOperationAction(ISD::MULHS, MVT::i64, Expand); + setOperationAction(ISD::MULHU, MVT::i64, + Subtarget->isVIS3() ? Legal : Expand); + setOperationAction(ISD::MULHS, MVT::i64, + Subtarget->isVIS3() ? Legal : Expand); setOperationAction(ISD::SHL_PARTS, MVT::i64, Expand); setOperationAction(ISD::SRA_PARTS, MVT::i64, Expand); diff --git a/llvm/lib/Target/Sparc/SparcInstrVIS.td b/llvm/lib/Target/Sparc/SparcInstrVIS.td index ee24d8a54fe8e..d9fe3b49821e5 100644 --- a/llvm/lib/Target/Sparc/SparcInstrVIS.td +++ b/llvm/lib/Target/Sparc/SparcInstrVIS.td @@ -294,4 +294,14 @@ def : Pat<(f32 fpnegimm0), (FNEGS (FZEROS))>; // VIS3 instruction patterns. let Predicates = [HasVIS3] in { def : Pat<(i64 (adde i64:$lhs, i64:$rhs)), (ADDXCCC $lhs, $rhs)>; + +def : Pat<(i64 (mulhu i64:$lhs, i64:$rhs)), (UMULXHI $lhs, $rhs)>; +// Signed "MULXHI". +// Based on the formula presented in OSA2011 §7.140, but with bitops to select +// the values to be added. +// TODO: This expansion should probably be moved to DAG legalization phase. +def : Pat<(i64 (mulhs i64:$lhs, i64:$rhs)), + (SUBrr (UMULXHI $lhs, $rhs), + (ADDrr (ANDrr (SRAXri $lhs, 63), $rhs), + (ANDrr (SRAXri $rhs, 63), $lhs)))>; } // Predicates = [HasVIS3] diff --git a/llvm/test/CodeGen/SPARC/multiply-extension.ll b/llvm/test/CodeGen/SPARC/multiply-extension.ll new file mode 100644 index 0000000000000..4d752ff101ca2 --- /dev/null +++ b/llvm/test/CodeGen/SPARC/multiply-extension.ll @@ -0,0 +1,59 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 +; RUN: llc < %s -mtriple=sparcv9 | FileCheck %s -check-prefix=V9 +; RUN: llc < %s -mtriple=sparcv9 -mattr=+vis3 | FileCheck %s -check-prefix=VIS3 + +define i128 @signed_multiply_extend(i64 %0, i64 %1) nounwind { +; V9-LABEL: signed_multiply_extend: +; V9: ! %bb.0: +; V9-NEXT: save %sp, -176, %sp +; V9-NEXT: srax %i0, 63, %o2 +; V9-NEXT: srax %i1, 63, %o0 +; V9-NEXT: mov %i1, %o1 +; V9-NEXT: call __multi3 +; V9-NEXT: mov %i0, %o3 +; V9-NEXT: mov %o0, %i0 +; V9-NEXT: ret +; V9-NEXT: restore %g0, %o1, %o1 +; +; VIS3-LABEL: signed_multiply_extend: +; VIS3: ! %bb.0: +; VIS3-NEXT: srax %o0, 63, %o2 +; VIS3-NEXT: and %o2, %o1, %o2 +; VIS3-NEXT: srax %o1, 63, %o3 +; VIS3-NEXT: and %o3, %o0, %o3 +; VIS3-NEXT: add %o3, %o2, %o2 +; VIS3-NEXT: umulxhi %o1, %o0, %o3 +; VIS3-NEXT: sub %o3, %o2, %o2 +; VIS3-NEXT: mulx %o1, %o0, %o1 +; VIS3-NEXT: retl +; VIS3-NEXT: mov %o2, %o0 + %3 = sext i64 %0 to i128 + %4 = sext i64 %1 to i128 + %5 = mul nsw i128 %4, %3 + ret i128 %5 +} + +define i128 @unsigned_multiply_extend(i64 %0, i64 %1) nounwind { +; V9-LABEL: unsigned_multiply_extend: +; V9: ! %bb.0: +; V9-NEXT: save %sp, -176, %sp +; V9-NEXT: mov %g0, %o0 +; V9-NEXT: mov %i1, %o1 +; V9-NEXT: mov %g0, %o2 +; V9-NEXT: call __multi3 +; V9-NEXT: mov %i0, %o3 +; V9-NEXT: mov %o0, %i0 +; V9-NEXT: ret +; V9-NEXT: restore %g0, %o1, %o1 +; +; VIS3-LABEL: unsigned_multiply_extend: +; VIS3: ! %bb.0: +; VIS3-NEXT: umulxhi %o1, %o0, %o2 +; VIS3-NEXT: mulx %o1, %o0, %o1 +; VIS3-NEXT: retl +; VIS3-NEXT: mov %o2, %o0 + %3 = zext i64 %0 to i128 + %4 = zext i64 %1 to i128 + %5 = mul nuw i128 %4, %3 + ret i128 %5 +} diff --git a/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll b/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll index 07e4c408a3ff0..1e5ab7922de08 100644 --- a/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll +++ b/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll @@ -1,6 +1,7 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py ; RUN: llc < %s -mtriple=sparc-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC ; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC64 +; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu -mattr=vis3 | FileCheck %s --check-prefixes=SPARC64-VIS3 define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC-LABEL: muloti_test: @@ -213,6 +214,49 @@ define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC64-NEXT: srl %i3, 0, %i2 ; SPARC64-NEXT: ret ; SPARC64-NEXT: restore +; +; SPARC64-VIS3-LABEL: muloti_test: +; SPARC64-VIS3: .register %g2, #scratch +; SPARC64-VIS3-NEXT: .register %g3, #scratch +; SPARC64-VIS3-NEXT: ! %bb.0: ! %start +; SPARC64-VIS3-NEXT: save %sp, -128, %sp +; SPARC64-VIS3-NEXT: mov %g0, %i5 +; SPARC64-VIS3-NEXT: umulxhi %i0, %i3, %i4 +; SPARC64-VIS3-NEXT: srax %i0, 63, %g2 +; SPARC64-VIS3-NEXT: mulx %g2, %i3, %g3 +; SPARC64-VIS3-NEXT: add %i4, %g3, %i4 +; SPARC64-VIS3-NEXT: umulxhi %i1, %i3, %g3 +; SPARC64-VIS3-NEXT: mulx %i0, %i3, %g4 +; SPARC64-VIS3-NEXT: addcc %g4, %g3, %g3 +; SPARC64-VIS3-NEXT: addxccc %i4, %g0, %g4 +; SPARC64-VIS3-NEXT: umulxhi %i1, %i2, %i4 +; SPARC64-VIS3-NEXT: srax %i2, 63, %g5 +; SPARC64-VIS3-NEXT: mulx %i1, %g5, %l0 +; SPARC64-VIS3-NEXT: add %i4, %l0, %l0 +; SPARC64-VIS3-NEXT: mulx %i1, %i2, %i4 +; SPARC64-VIS3-NEXT: addcc %i4, %g3, %i4 +; SPARC64-VIS3-NEXT: addxccc %l0, %g0, %g3 +; SPARC64-VIS3-NEXT: srax %g3, 63, %l0 +; SPARC64-VIS3-NEXT: addcc %g4, %g3, %g3 +; SPARC64-VIS3-NEXT: srax %g4, 63, %g4 +; SPARC64-VIS3-NEXT: addxccc %g4, %l0, %g4 +; SPARC64-VIS3-NEXT: and %g5, %i0, %g5 +; SPARC64-VIS3-NEXT: and %g2, %i2, %g2 +; SPARC64-VIS3-NEXT: add %g2, %g5, %g2 +; SPARC64-VIS3-NEXT: umulxhi %i0, %i2, %g5 +; SPARC64-VIS3-NEXT: sub %g5, %g2, %g2 +; SPARC64-VIS3-NEXT: mulx %i0, %i2, %i0 +; SPARC64-VIS3-NEXT: addcc %i0, %g3, %i0 +; SPARC64-VIS3-NEXT: addxccc %g2, %g4, %i2 +; SPARC64-VIS3-NEXT: srax %i4, 63, %g2 +; SPARC64-VIS3-NEXT: xor %i2, %g2, %i2 +; SPARC64-VIS3-NEXT: xor %i0, %g2, %i0 +; SPARC64-VIS3-NEXT: or %i0, %i2, %i0 +; SPARC64-VIS3-NEXT: movrnz %i0, 1, %i5 +; SPARC64-VIS3-NEXT: mulx %i1, %i3, %i1 +; SPARC64-VIS3-NEXT: srl %i5, 0, %i2 +; SPARC64-VIS3-NEXT: ret +; SPARC64-VIS3-NEXT: restore %g0, %i4, %o0 start: %0 = tail call { i128, i1 } @llvm.smul.with.overflow.i128(i128 %l, i128 %r) %1 = extractvalue { i128, i1 } %0, 0 diff --git a/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll b/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll index f3835790210a0..6d197c88bfecd 100644 --- a/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll +++ b/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll @@ -1,6 +1,7 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py ; RUN: llc < %s -mtriple=sparc-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC ; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC64 +; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu -mattr=vis3 | FileCheck %s --check-prefixes=SPARC64-VIS3 define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC-LABEL: muloti_test: @@ -199,6 +200,38 @@ define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC64-NEXT: srl %i1, 0, %i2 ; SPARC64-NEXT: ret ; SPARC64-NEXT: restore %g0, %o1, %o1 +; +; SPARC64-VIS3-LABEL: muloti_test: +; SPARC64-VIS3: .register %g2, #scratch +; SPARC64-VIS3-NEXT: .register %g3, #scratch +; SPARC64-VIS3-NEXT: ! %bb.0: ! %start +; SPARC64-VIS3-NEXT: save %sp, -128, %sp +; SPARC64-VIS3-NEXT: mov %g0, %i5 +; SPARC64-VIS3-NEXT: mov %g0, %g2 +; SPARC64-VIS3-NEXT: mov %g0, %g3 +; SPARC64-VIS3-NEXT: mov %g0, %g4 +; SPARC64-VIS3-NEXT: mov %g0, %g5 +; SPARC64-VIS3-NEXT: mulx %i2, %i1, %i4 +; SPARC64-VIS3-NEXT: mulx %i0, %i3, %l0 +; SPARC64-VIS3-NEXT: add %l0, %i4, %i4 +; SPARC64-VIS3-NEXT: umulxhi %i1, %i3, %l0 +; SPARC64-VIS3-NEXT: add %l0, %i4, %i4 +; SPARC64-VIS3-NEXT: cmp %i4, %l0 +; SPARC64-VIS3-NEXT: movrnz %i2, 1, %g2 +; SPARC64-VIS3-NEXT: movrnz %i0, 1, %g3 +; SPARC64-VIS3-NEXT: and %g3, %g2, %g2 +; SPARC64-VIS3-NEXT: umulxhi %i0, %i3, %i0 +; SPARC64-VIS3-NEXT: movrnz %i0, 1, %g4 +; SPARC64-VIS3-NEXT: movcs %xcc, 1, %i5 +; SPARC64-VIS3-NEXT: or %g2, %g4, %i0 +; SPARC64-VIS3-NEXT: umulxhi %i2, %i1, %i2 +; SPARC64-VIS3-NEXT: movrnz %i2, 1, %g5 +; SPARC64-VIS3-NEXT: or %i0, %g5, %i0 +; SPARC64-VIS3-NEXT: or %i0, %i5, %i0 +; SPARC64-VIS3-NEXT: mulx %i1, %i3, %i1 +; SPARC64-VIS3-NEXT: srl %i0, 0, %i2 +; SPARC64-VIS3-NEXT: ret +; SPARC64-VIS3-NEXT: restore %g0, %i4, %o0 start: %0 = tail call { i128, i1 } @llvm.umul.with.overflow.i128(i128 %l, i128 %r) %1 = extractvalue { i128, i1 } %0, 0 From 0439a4eca78fa1e3aa45b49ff349c3da4fb02f48 Mon Sep 17 00:00:00 2001 From: Jim Lin Date: Wed, 16 Apr 2025 10:16:31 +0800 Subject: [PATCH 54/63] [RISCV] Add new CondCode COND_CV_BEQIMM/COND_CV_BNEIMM for CV immediate branch (#135771) If there is another branch instruction also with immediate operand, but it is used to specify which bit to be tested is set or clear. We only check whether operand2 is immediate or not here. There are no way to distinguish between them. So add new CondCode COND_CV_BEQIMM/COND_CV_BNEIMM that we can know what kinds of immediate branch instruction are matched in Select_* Pseudo. --- .../RISCV/GISel/RISCVInstructionSelector.cpp | 2 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 2 +- llvm/lib/Target/RISCV/RISCVInstrInfo.cpp | 42 +++++++++---------- llvm/lib/Target/RISCV/RISCVInstrInfo.h | 6 ++- llvm/lib/Target/RISCV/RISCVInstrInfoXCV.td | 9 +++- 5 files changed, 33 insertions(+), 28 deletions(-) diff --git a/llvm/lib/Target/RISCV/GISel/RISCVInstructionSelector.cpp b/llvm/lib/Target/RISCV/GISel/RISCVInstructionSelector.cpp index 18ce5407f816c..f83c2b6da8923 100644 --- a/llvm/lib/Target/RISCV/GISel/RISCVInstructionSelector.cpp +++ b/llvm/lib/Target/RISCV/GISel/RISCVInstructionSelector.cpp @@ -789,7 +789,7 @@ bool RISCVInstructionSelector::select(MachineInstr &MI) { RISCVCC::CondCode CC; getOperandsForBranch(MI.getOperand(0).getReg(), CC, LHS, RHS, *MRI); - auto Bcc = MIB.buildInstr(RISCVCC::getBrCond(STI, CC), {}, {LHS, RHS}) + auto Bcc = MIB.buildInstr(RISCVCC::getBrCond(CC), {}, {LHS, RHS}) .addMBB(MI.getOperand(1).getMBB()); MI.eraseFromParent(); return constrainSelectedInstRegOperands(*Bcc, TII, TRI, RBI); diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp index f24752b8721f5..beea13d5f8f3e 100644 --- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp @@ -20645,7 +20645,7 @@ static MachineBasicBlock *emitSelectPseudo(MachineInstr &MI, // Insert appropriate branch. if (MI.getOperand(2).isImm()) - BuildMI(HeadMBB, DL, TII.getBrCond(CC, MI.getOperand(2).isImm())) + BuildMI(HeadMBB, DL, TII.getBrCond(CC)) .addReg(LHS) .addImm(MI.getOperand(2).getImm()) .addMBB(TailMBB); diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp b/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp index f8a35533ba952..5d661a3438b1c 100644 --- a/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp +++ b/llvm/lib/Target/RISCV/RISCVInstrInfo.cpp @@ -974,10 +974,6 @@ static RISCVCC::CondCode getCondFromBranchOpc(unsigned Opc) { switch (Opc) { default: return RISCVCC::COND_INVALID; - case RISCV::CV_BEQIMM: - return RISCVCC::COND_EQ; - case RISCV::CV_BNEIMM: - return RISCVCC::COND_NE; case RISCV::BEQ: return RISCVCC::COND_EQ; case RISCV::BNE: @@ -990,6 +986,10 @@ static RISCVCC::CondCode getCondFromBranchOpc(unsigned Opc) { return RISCVCC::COND_LTU; case RISCV::BGEU: return RISCVCC::COND_GEU; + case RISCV::CV_BEQIMM: + return RISCVCC::COND_CV_BEQIMM; + case RISCV::CV_BNEIMM: + return RISCVCC::COND_CV_BNEIMM; } } @@ -1027,23 +1027,14 @@ static void parseCondBranch(MachineInstr &LastInst, MachineBasicBlock *&Target, Cond.push_back(LastInst.getOperand(1)); } -unsigned RISCVCC::getBrCond(const RISCVSubtarget &STI, RISCVCC::CondCode CC, - bool Imm) { +unsigned RISCVCC::getBrCond(RISCVCC::CondCode CC) { switch (CC) { default: llvm_unreachable("Unknown condition code!"); case RISCVCC::COND_EQ: - if (!Imm) - return RISCV::BEQ; - if (STI.hasVendorXCVbi()) - return RISCV::CV_BEQIMM; - llvm_unreachable("Unknown branch immediate!"); + return RISCV::BEQ; case RISCVCC::COND_NE: - if (!Imm) - return RISCV::BNE; - if (STI.hasVendorXCVbi()) - return RISCV::CV_BNEIMM; - llvm_unreachable("Unknown branch immediate!"); + return RISCV::BNE; case RISCVCC::COND_LT: return RISCV::BLT; case RISCVCC::COND_GE: @@ -1052,12 +1043,15 @@ unsigned RISCVCC::getBrCond(const RISCVSubtarget &STI, RISCVCC::CondCode CC, return RISCV::BLTU; case RISCVCC::COND_GEU: return RISCV::BGEU; + case RISCVCC::COND_CV_BEQIMM: + return RISCV::CV_BEQIMM; + case RISCVCC::COND_CV_BNEIMM: + return RISCV::CV_BNEIMM; } } -const MCInstrDesc &RISCVInstrInfo::getBrCond(RISCVCC::CondCode CC, - bool Imm) const { - return get(RISCVCC::getBrCond(STI, CC, Imm)); +const MCInstrDesc &RISCVInstrInfo::getBrCond(RISCVCC::CondCode CC) const { + return get(RISCVCC::getBrCond(CC)); } RISCVCC::CondCode RISCVCC::getOppositeBranchCondition(RISCVCC::CondCode CC) { @@ -1076,6 +1070,10 @@ RISCVCC::CondCode RISCVCC::getOppositeBranchCondition(RISCVCC::CondCode CC) { return RISCVCC::COND_GEU; case RISCVCC::COND_GEU: return RISCVCC::COND_LTU; + case RISCVCC::COND_CV_BEQIMM: + return RISCVCC::COND_CV_BNEIMM; + case RISCVCC::COND_CV_BNEIMM: + return RISCVCC::COND_CV_BEQIMM; } } @@ -1206,10 +1204,8 @@ unsigned RISCVInstrInfo::insertBranch( // Either a one or two-way conditional branch. auto CC = static_cast(Cond[0].getImm()); - MachineInstr &CondMI = *BuildMI(&MBB, DL, getBrCond(CC, Cond[2].isImm())) - .add(Cond[1]) - .add(Cond[2]) - .addMBB(TBB); + MachineInstr &CondMI = + *BuildMI(&MBB, DL, getBrCond(CC)).add(Cond[1]).add(Cond[2]).addMBB(TBB); if (BytesAdded) *BytesAdded += getInstSizeInBytes(CondMI); diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfo.h b/llvm/lib/Target/RISCV/RISCVInstrInfo.h index bf0d6b2c59a45..67e457d64f6e3 100644 --- a/llvm/lib/Target/RISCV/RISCVInstrInfo.h +++ b/llvm/lib/Target/RISCV/RISCVInstrInfo.h @@ -41,11 +41,13 @@ enum CondCode { COND_GE, COND_LTU, COND_GEU, + COND_CV_BEQIMM, + COND_CV_BNEIMM, COND_INVALID }; CondCode getOppositeBranchCondition(CondCode); -unsigned getBrCond(const RISCVSubtarget &STI, CondCode CC, bool Imm = false); +unsigned getBrCond(CondCode CC); } // end of namespace RISCVCC @@ -65,7 +67,7 @@ class RISCVInstrInfo : public RISCVGenInstrInfo { explicit RISCVInstrInfo(RISCVSubtarget &STI); MCInst getNop() const override; - const MCInstrDesc &getBrCond(RISCVCC::CondCode CC, bool Imm = false) const; + const MCInstrDesc &getBrCond(RISCVCC::CondCode CC) const; Register isLoadFromStackSlot(const MachineInstr &MI, int &FrameIndex) const override; diff --git a/llvm/lib/Target/RISCV/RISCVInstrInfoXCV.td b/llvm/lib/Target/RISCV/RISCVInstrInfoXCV.td index b5df7b54fc9f1..5ce08d64b141d 100644 --- a/llvm/lib/Target/RISCV/RISCVInstrInfoXCV.td +++ b/llvm/lib/Target/RISCV/RISCVInstrInfoXCV.td @@ -791,6 +791,13 @@ let Predicates = [HasVendorXCValu, IsRV32], AddedComplexity = 1 in { // Patterns for immediate branching operations //===----------------------------------------------------------------------===// +def IntCCtoRISCVCCCV : SDNodeXForm(N->getOperand(2))->get(); + assert(CC == ISD::SETEQ || CC == ISD::SETNE); + RISCVCC::CondCode BrCC = CC == ISD::SETEQ ? RISCVCC::COND_CV_BEQIMM : RISCVCC::COND_CV_BNEIMM; + return CurDAG->getTargetConstant(BrCC, SDLoc(N), Subtarget->getXLenVT()); +}]>; + let Predicates = [HasVendorXCVbi, IsRV32], AddedComplexity = 2 in { def : Pat<(riscv_brcc GPR:$rs1, simm5:$imm5, SETEQ, bb:$imm12), (CV_BEQIMM GPR:$rs1, simm5:$imm5, bare_simm13_lsb0:$imm12)>; @@ -807,7 +814,7 @@ let Predicates = [HasVendorXCVbi, IsRV32], AddedComplexity = 2 in { : Pat<(riscv_selectcc_frag:$cc (i32 GPR:$lhs), simm5:$Constant, Cond, (i32 GPR:$truev), GPR:$falsev), (Select_GPR_Using_CC_Imm GPR:$lhs, simm5:$Constant, - (IntCCtoRISCVCC $cc), GPR:$truev, GPR:$falsev)>; + (IntCCtoRISCVCCCV $cc), GPR:$truev, GPR:$falsev)>; def : Selectbi; def : Selectbi; From 5e9650ec2deb2f2bb6d5ad28e83bb6cd3c4189e4 Mon Sep 17 00:00:00 2001 From: Koakuma Date: Wed, 16 Apr 2025 09:27:17 +0700 Subject: [PATCH 55/63] Revert "[SPARC] Use umulxhi to do extending 64x64->128 multiply when we have VIS3" (#135897) This change breaks multiply tests on SPARC. https://lab.llvm.org/buildbot/#/builders/108/builds/11691/steps/6/logs/FAIL__LLVM__multiply-extension_ll Reverts llvm/llvm-project#135714 --- llvm/lib/Target/Sparc/SparcISelLowering.cpp | 6 +- llvm/lib/Target/Sparc/SparcInstrVIS.td | 10 ---- llvm/test/CodeGen/SPARC/multiply-extension.ll | 59 ------------------- .../SPARC/smulo-128-legalisation-lowering.ll | 44 -------------- .../SPARC/umulo-128-legalisation-lowering.ll | 33 ----------- 5 files changed, 2 insertions(+), 150 deletions(-) delete mode 100644 llvm/test/CodeGen/SPARC/multiply-extension.ll diff --git a/llvm/lib/Target/Sparc/SparcISelLowering.cpp b/llvm/lib/Target/Sparc/SparcISelLowering.cpp index 0ad261135651f..098e5f22834f4 100644 --- a/llvm/lib/Target/Sparc/SparcISelLowering.cpp +++ b/llvm/lib/Target/Sparc/SparcISelLowering.cpp @@ -1854,10 +1854,8 @@ SparcTargetLowering::SparcTargetLowering(const TargetMachine &TM, if (Subtarget->is64Bit()) { setOperationAction(ISD::UMUL_LOHI, MVT::i64, Expand); setOperationAction(ISD::SMUL_LOHI, MVT::i64, Expand); - setOperationAction(ISD::MULHU, MVT::i64, - Subtarget->isVIS3() ? Legal : Expand); - setOperationAction(ISD::MULHS, MVT::i64, - Subtarget->isVIS3() ? Legal : Expand); + setOperationAction(ISD::MULHU, MVT::i64, Expand); + setOperationAction(ISD::MULHS, MVT::i64, Expand); setOperationAction(ISD::SHL_PARTS, MVT::i64, Expand); setOperationAction(ISD::SRA_PARTS, MVT::i64, Expand); diff --git a/llvm/lib/Target/Sparc/SparcInstrVIS.td b/llvm/lib/Target/Sparc/SparcInstrVIS.td index d9fe3b49821e5..ee24d8a54fe8e 100644 --- a/llvm/lib/Target/Sparc/SparcInstrVIS.td +++ b/llvm/lib/Target/Sparc/SparcInstrVIS.td @@ -294,14 +294,4 @@ def : Pat<(f32 fpnegimm0), (FNEGS (FZEROS))>; // VIS3 instruction patterns. let Predicates = [HasVIS3] in { def : Pat<(i64 (adde i64:$lhs, i64:$rhs)), (ADDXCCC $lhs, $rhs)>; - -def : Pat<(i64 (mulhu i64:$lhs, i64:$rhs)), (UMULXHI $lhs, $rhs)>; -// Signed "MULXHI". -// Based on the formula presented in OSA2011 §7.140, but with bitops to select -// the values to be added. -// TODO: This expansion should probably be moved to DAG legalization phase. -def : Pat<(i64 (mulhs i64:$lhs, i64:$rhs)), - (SUBrr (UMULXHI $lhs, $rhs), - (ADDrr (ANDrr (SRAXri $lhs, 63), $rhs), - (ANDrr (SRAXri $rhs, 63), $lhs)))>; } // Predicates = [HasVIS3] diff --git a/llvm/test/CodeGen/SPARC/multiply-extension.ll b/llvm/test/CodeGen/SPARC/multiply-extension.ll deleted file mode 100644 index 4d752ff101ca2..0000000000000 --- a/llvm/test/CodeGen/SPARC/multiply-extension.ll +++ /dev/null @@ -1,59 +0,0 @@ -; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 -; RUN: llc < %s -mtriple=sparcv9 | FileCheck %s -check-prefix=V9 -; RUN: llc < %s -mtriple=sparcv9 -mattr=+vis3 | FileCheck %s -check-prefix=VIS3 - -define i128 @signed_multiply_extend(i64 %0, i64 %1) nounwind { -; V9-LABEL: signed_multiply_extend: -; V9: ! %bb.0: -; V9-NEXT: save %sp, -176, %sp -; V9-NEXT: srax %i0, 63, %o2 -; V9-NEXT: srax %i1, 63, %o0 -; V9-NEXT: mov %i1, %o1 -; V9-NEXT: call __multi3 -; V9-NEXT: mov %i0, %o3 -; V9-NEXT: mov %o0, %i0 -; V9-NEXT: ret -; V9-NEXT: restore %g0, %o1, %o1 -; -; VIS3-LABEL: signed_multiply_extend: -; VIS3: ! %bb.0: -; VIS3-NEXT: srax %o0, 63, %o2 -; VIS3-NEXT: and %o2, %o1, %o2 -; VIS3-NEXT: srax %o1, 63, %o3 -; VIS3-NEXT: and %o3, %o0, %o3 -; VIS3-NEXT: add %o3, %o2, %o2 -; VIS3-NEXT: umulxhi %o1, %o0, %o3 -; VIS3-NEXT: sub %o3, %o2, %o2 -; VIS3-NEXT: mulx %o1, %o0, %o1 -; VIS3-NEXT: retl -; VIS3-NEXT: mov %o2, %o0 - %3 = sext i64 %0 to i128 - %4 = sext i64 %1 to i128 - %5 = mul nsw i128 %4, %3 - ret i128 %5 -} - -define i128 @unsigned_multiply_extend(i64 %0, i64 %1) nounwind { -; V9-LABEL: unsigned_multiply_extend: -; V9: ! %bb.0: -; V9-NEXT: save %sp, -176, %sp -; V9-NEXT: mov %g0, %o0 -; V9-NEXT: mov %i1, %o1 -; V9-NEXT: mov %g0, %o2 -; V9-NEXT: call __multi3 -; V9-NEXT: mov %i0, %o3 -; V9-NEXT: mov %o0, %i0 -; V9-NEXT: ret -; V9-NEXT: restore %g0, %o1, %o1 -; -; VIS3-LABEL: unsigned_multiply_extend: -; VIS3: ! %bb.0: -; VIS3-NEXT: umulxhi %o1, %o0, %o2 -; VIS3-NEXT: mulx %o1, %o0, %o1 -; VIS3-NEXT: retl -; VIS3-NEXT: mov %o2, %o0 - %3 = zext i64 %0 to i128 - %4 = zext i64 %1 to i128 - %5 = mul nuw i128 %4, %3 - ret i128 %5 -} diff --git a/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll b/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll index 1e5ab7922de08..07e4c408a3ff0 100644 --- a/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll +++ b/llvm/test/CodeGen/SPARC/smulo-128-legalisation-lowering.ll @@ -1,7 +1,6 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py ; RUN: llc < %s -mtriple=sparc-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC ; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC64 -; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu -mattr=vis3 | FileCheck %s --check-prefixes=SPARC64-VIS3 define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC-LABEL: muloti_test: @@ -214,49 +213,6 @@ define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC64-NEXT: srl %i3, 0, %i2 ; SPARC64-NEXT: ret ; SPARC64-NEXT: restore -; -; SPARC64-VIS3-LABEL: muloti_test: -; SPARC64-VIS3: .register %g2, #scratch -; SPARC64-VIS3-NEXT: .register %g3, #scratch -; SPARC64-VIS3-NEXT: ! %bb.0: ! %start -; SPARC64-VIS3-NEXT: save %sp, -128, %sp -; SPARC64-VIS3-NEXT: mov %g0, %i5 -; SPARC64-VIS3-NEXT: umulxhi %i0, %i3, %i4 -; SPARC64-VIS3-NEXT: srax %i0, 63, %g2 -; SPARC64-VIS3-NEXT: mulx %g2, %i3, %g3 -; SPARC64-VIS3-NEXT: add %i4, %g3, %i4 -; SPARC64-VIS3-NEXT: umulxhi %i1, %i3, %g3 -; SPARC64-VIS3-NEXT: mulx %i0, %i3, %g4 -; SPARC64-VIS3-NEXT: addcc %g4, %g3, %g3 -; SPARC64-VIS3-NEXT: addxccc %i4, %g0, %g4 -; SPARC64-VIS3-NEXT: umulxhi %i1, %i2, %i4 -; SPARC64-VIS3-NEXT: srax %i2, 63, %g5 -; SPARC64-VIS3-NEXT: mulx %i1, %g5, %l0 -; SPARC64-VIS3-NEXT: add %i4, %l0, %l0 -; SPARC64-VIS3-NEXT: mulx %i1, %i2, %i4 -; SPARC64-VIS3-NEXT: addcc %i4, %g3, %i4 -; SPARC64-VIS3-NEXT: addxccc %l0, %g0, %g3 -; SPARC64-VIS3-NEXT: srax %g3, 63, %l0 -; SPARC64-VIS3-NEXT: addcc %g4, %g3, %g3 -; SPARC64-VIS3-NEXT: srax %g4, 63, %g4 -; SPARC64-VIS3-NEXT: addxccc %g4, %l0, %g4 -; SPARC64-VIS3-NEXT: and %g5, %i0, %g5 -; SPARC64-VIS3-NEXT: and %g2, %i2, %g2 -; SPARC64-VIS3-NEXT: add %g2, %g5, %g2 -; SPARC64-VIS3-NEXT: umulxhi %i0, %i2, %g5 -; SPARC64-VIS3-NEXT: sub %g5, %g2, %g2 -; SPARC64-VIS3-NEXT: mulx %i0, %i2, %i0 -; SPARC64-VIS3-NEXT: addcc %i0, %g3, %i0 -; SPARC64-VIS3-NEXT: addxccc %g2, %g4, %i2 -; SPARC64-VIS3-NEXT: srax %i4, 63, %g2 -; SPARC64-VIS3-NEXT: xor %i2, %g2, %i2 -; SPARC64-VIS3-NEXT: xor %i0, %g2, %i0 -; SPARC64-VIS3-NEXT: or %i0, %i2, %i0 -; SPARC64-VIS3-NEXT: movrnz %i0, 1, %i5 -; SPARC64-VIS3-NEXT: mulx %i1, %i3, %i1 -; SPARC64-VIS3-NEXT: srl %i5, 0, %i2 -; SPARC64-VIS3-NEXT: ret -; SPARC64-VIS3-NEXT: restore %g0, %i4, %o0 start: %0 = tail call { i128, i1 } @llvm.smul.with.overflow.i128(i128 %l, i128 %r) %1 = extractvalue { i128, i1 } %0, 0 diff --git a/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll b/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll index 6d197c88bfecd..f3835790210a0 100644 --- a/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll +++ b/llvm/test/CodeGen/SPARC/umulo-128-legalisation-lowering.ll @@ -1,7 +1,6 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py ; RUN: llc < %s -mtriple=sparc-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC ; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu | FileCheck %s --check-prefixes=SPARC64 -; RUN: llc < %s -mtriple=sparc64-unknown-linux-gnu -mattr=vis3 | FileCheck %s --check-prefixes=SPARC64-VIS3 define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC-LABEL: muloti_test: @@ -200,38 +199,6 @@ define { i128, i8 } @muloti_test(i128 %l, i128 %r) nounwind { ; SPARC64-NEXT: srl %i1, 0, %i2 ; SPARC64-NEXT: ret ; SPARC64-NEXT: restore %g0, %o1, %o1 -; -; SPARC64-VIS3-LABEL: muloti_test: -; SPARC64-VIS3: .register %g2, #scratch -; SPARC64-VIS3-NEXT: .register %g3, #scratch -; SPARC64-VIS3-NEXT: ! %bb.0: ! %start -; SPARC64-VIS3-NEXT: save %sp, -128, %sp -; SPARC64-VIS3-NEXT: mov %g0, %i5 -; SPARC64-VIS3-NEXT: mov %g0, %g2 -; SPARC64-VIS3-NEXT: mov %g0, %g3 -; SPARC64-VIS3-NEXT: mov %g0, %g4 -; SPARC64-VIS3-NEXT: mov %g0, %g5 -; SPARC64-VIS3-NEXT: mulx %i2, %i1, %i4 -; SPARC64-VIS3-NEXT: mulx %i0, %i3, %l0 -; SPARC64-VIS3-NEXT: add %l0, %i4, %i4 -; SPARC64-VIS3-NEXT: umulxhi %i1, %i3, %l0 -; SPARC64-VIS3-NEXT: add %l0, %i4, %i4 -; SPARC64-VIS3-NEXT: cmp %i4, %l0 -; SPARC64-VIS3-NEXT: movrnz %i2, 1, %g2 -; SPARC64-VIS3-NEXT: movrnz %i0, 1, %g3 -; SPARC64-VIS3-NEXT: and %g3, %g2, %g2 -; SPARC64-VIS3-NEXT: umulxhi %i0, %i3, %i0 -; SPARC64-VIS3-NEXT: movrnz %i0, 1, %g4 -; SPARC64-VIS3-NEXT: movcs %xcc, 1, %i5 -; SPARC64-VIS3-NEXT: or %g2, %g4, %i0 -; SPARC64-VIS3-NEXT: umulxhi %i2, %i1, %i2 -; SPARC64-VIS3-NEXT: movrnz %i2, 1, %g5 -; SPARC64-VIS3-NEXT: or %i0, %g5, %i0 -; SPARC64-VIS3-NEXT: or %i0, %i5, %i0 -; SPARC64-VIS3-NEXT: mulx %i1, %i3, %i1 -; SPARC64-VIS3-NEXT: srl %i0, 0, %i2 -; SPARC64-VIS3-NEXT: ret -; SPARC64-VIS3-NEXT: restore %g0, %i4, %o0 start: %0 = tail call { i128, i1 } @llvm.umul.with.overflow.i128(i128 %l, i128 %r) %1 = extractvalue { i128, i1 } %0, 0 From bed03ae36600f83f214c41af333f47fe8ead9ede Mon Sep 17 00:00:00 2001 From: Feng Zou Date: Wed, 16 Apr 2025 10:47:05 +0800 Subject: [PATCH 56/63] [X86] Fix the issue of creating index reg negations (#135632) The 8 and 16 bit LEA instruction support was added by PR #122102, and we have to update creating index register negations accordingly. The issue is exposed with APX NDD instructions. --- llvm/lib/Target/X86/X86ISelDAGToDAG.cpp | 19 ++++- .../CodeGen/X86/apx/ndd-neg-addr-index.ll | 71 +++++++++++++++++++ 2 files changed, 88 insertions(+), 2 deletions(-) create mode 100644 llvm/test/CodeGen/X86/apx/ndd-neg-addr-index.ll diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp index d322e70fc0c20..01118beb9cf5e 100644 --- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp +++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp @@ -275,8 +275,23 @@ namespace { #define GET_ND_IF_ENABLED(OPC) (Subtarget->hasNDD() ? OPC##_ND : OPC) // Negate the index if needed. if (AM.NegateIndex) { - unsigned NegOpc = VT == MVT::i64 ? GET_ND_IF_ENABLED(X86::NEG64r) - : GET_ND_IF_ENABLED(X86::NEG32r); + unsigned NegOpc; + switch (VT.SimpleTy) { + default: + llvm_unreachable("Unsupported VT!"); + case MVT::i64: + NegOpc = GET_ND_IF_ENABLED(X86::NEG64r); + break; + case MVT::i32: + NegOpc = GET_ND_IF_ENABLED(X86::NEG32r); + break; + case MVT::i16: + NegOpc = GET_ND_IF_ENABLED(X86::NEG16r); + break; + case MVT::i8: + NegOpc = GET_ND_IF_ENABLED(X86::NEG8r); + break; + } SDValue Neg = SDValue(CurDAG->getMachineNode(NegOpc, DL, VT, MVT::i32, AM.IndexReg), 0); AM.IndexReg = Neg; diff --git a/llvm/test/CodeGen/X86/apx/ndd-neg-addr-index.ll b/llvm/test/CodeGen/X86/apx/ndd-neg-addr-index.ll new file mode 100644 index 0000000000000..6679b5f58e8c1 --- /dev/null +++ b/llvm/test/CodeGen/X86/apx/ndd-neg-addr-index.ll @@ -0,0 +1,71 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 +; RUN: llc %s -mtriple=x86_64-unknown -mattr=+ndd -verify-machineinstrs --show-mc-encoding -o - | FileCheck %s --check-prefix=NDD + + +define void @neg_8bit_1(i1 %cmp) { +; NDD-LABEL: neg_8bit_1: +; NDD: # %bb.0: # %entry +; NDD-NEXT: andb $1, %dil, %al # encoding: [0x62,0xf4,0x7c,0x18,0x80,0xe7,0x01] +; NDD-NEXT: movzbl 0, %ecx # encoding: [0x0f,0xb6,0x0c,0x25,0x00,0x00,0x00,0x00] +; NDD-NEXT: negb %al, %al # encoding: [0x62,0xf4,0x7c,0x18,0xf6,0xd8] +; NDD-NEXT: leab 2(%rcx,%rax), %al # encoding: [0x66,0x8d,0x44,0x01,0x02] +; NDD-NEXT: movb %al, 0 # encoding: [0x88,0x04,0x25,0x00,0x00,0x00,0x00] +; NDD-NEXT: retq # encoding: [0xc3] +entry: + %cond = select i1 %cmp, i8 1, i8 2 + %0 = load i8, ptr null, align 4 + %add = add i8 %cond, %0 + store i8 %add, ptr null, align 4 + ret void +} + +define void @neg_8bit_2(i8 %int8) { +; NDD-LABEL: neg_8bit_2: +; NDD: # %bb.0: # %entry +; NDD-NEXT: # kill: def $edi killed $edi def $rdi +; NDD-NEXT: addb %dil, %dil, %al # encoding: [0x62,0xf4,0x7c,0x18,0x00,0xff] +; NDD-NEXT: negb %al, %al # encoding: [0x62,0xf4,0x7c,0x18,0xf6,0xd8] +; NDD-NEXT: leab 1(%rdi,%rax), %al # encoding: [0x66,0x8d,0x44,0x07,0x01] +; NDD-NEXT: mulb %dil # encoding: [0x40,0xf6,0xe7] +; NDD-NEXT: testb %al, %al # encoding: [0x84,0xc0] +; NDD-NEXT: retq # encoding: [0xc3] +entry: + %0 = shl i8 %int8, 1 + %sub = sub i8 %int8, %0 + %add = add i8 %sub, 1 + %div = mul i8 %add, %int8 + %cmp = icmp slt i8 %div, 0 + br i1 %cmp, label %label2, label %label1 + +label1: ; preds = %entry + ret void + +label2: ; preds = %entry + ret void +} + +define i32 @neg_16bit(i16 %0) { +; NDD-LABEL: neg_16bit: +; NDD: # %bb.0: # %entry +; NDD-NEXT: # kill: def $edi killed $edi def $rdi +; NDD-NEXT: incw %di, %ax # encoding: [0x62,0xf4,0x7d,0x18,0xff,0xc7] +; NDD-NEXT: addw $256, %di, %cx # encoding: [0x62,0xf4,0x75,0x18,0x81,0xc7,0x00,0x01] +; NDD-NEXT: # imm = 0x100 +; NDD-NEXT: testw %ax, %ax # encoding: [0x66,0x85,0xc0] +; NDD-NEXT: cmovsl %ecx, %eax # EVEX TO LEGACY Compression encoding: [0x0f,0x48,0xc1] +; NDD-NEXT: andw $-256, %ax # EVEX TO LEGACY Compression encoding: [0x66,0x25,0x00,0xff] +; NDD-NEXT: negw %ax, %ax # encoding: [0x62,0xf4,0x7d,0x18,0xf7,0xd8] +; NDD-NEXT: leaw 1(%rdi,%rax), %ax # encoding: [0x66,0x8d,0x44,0x07,0x01] +; NDD-NEXT: movzwl %ax, %eax # encoding: [0x0f,0xb7,0xc0] +; NDD-NEXT: movq %rax, 0 # encoding: [0x48,0x89,0x04,0x25,0x00,0x00,0x00,0x00] +; NDD-NEXT: xorl %eax, %eax # encoding: [0x31,0xc0] +; NDD-NEXT: retq # encoding: [0xc3] +entry: + %add = add i16 %0, 1 + %rem = srem i16 %add, 256 + %1 = zext i16 %rem to i19 + %2 = sext i19 %1 to i64 + %3 = getelementptr i8, ptr null, i64 %2 + store ptr %3, ptr null, align 4 + ret i32 0 +} From e676866368a84c88aad90e138268e00a1c56a230 Mon Sep 17 00:00:00 2001 From: yingopq <115543042+yingopq@users.noreply.github.com> Date: Wed, 16 Apr 2025 10:56:06 +0800 Subject: [PATCH 57/63] [Mips] Fix clang crashes when compiling a variadic function while targeting mips3 (#130558) issue reason: Because mips3 has the feature 'FeatureGP64Bit', when target mips3 process function `writeVarArgRegs`, the result of `getGPRSizeInBytes` is 8 and the result of `GetVarArgRegs` is `Mips::A0, Mips::A1, Mips::A2, Mips::A3`. This would generate `gpr64 = COPY $a1` which should be `gpr64 = COPY $a1_64`. Also when process `CC_Mips_FixedArg`, mips would CCDelegateTo `CC_MipsO32_FP`. In fact, it should CCDelegateTo `CC_MipsN`. Fix #98716. --- .../Target/Mips/MCTargetDesc/MipsABIInfo.cpp | 10 ++-- .../Target/Mips/MCTargetDesc/MipsABIInfo.h | 2 +- llvm/lib/Target/Mips/MipsCallLowering.cpp | 3 +- llvm/lib/Target/Mips/MipsCallingConv.td | 2 +- llvm/lib/Target/Mips/MipsISelLowering.cpp | 3 +- llvm/test/CodeGen/Mips/vararg.ll | 54 +++++++++++++++++++ 6 files changed, 66 insertions(+), 8 deletions(-) create mode 100644 llvm/test/CodeGen/Mips/vararg.ll diff --git a/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.cpp b/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.cpp index 08cbba952ccc8..1be29cf3c94b9 100644 --- a/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.cpp +++ b/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.cpp @@ -37,9 +37,13 @@ ArrayRef MipsABIInfo::GetByValArgRegs() const { llvm_unreachable("Unhandled ABI"); } -ArrayRef MipsABIInfo::GetVarArgRegs() const { - if (IsO32()) - return ArrayRef(O32IntRegs); +ArrayRef MipsABIInfo::getVarArgRegs(bool isGP64bit) const { + if (IsO32()) { + if (isGP64bit) + return ArrayRef(Mips64IntRegs); + else + return ArrayRef(O32IntRegs); + } if (IsN32() || IsN64()) return ArrayRef(Mips64IntRegs); llvm_unreachable("Unhandled ABI"); diff --git a/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.h b/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.h index 41f80771142de..44b023c7c3ef6 100644 --- a/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.h +++ b/llvm/lib/Target/Mips/MCTargetDesc/MipsABIInfo.h @@ -46,7 +46,7 @@ class MipsABIInfo { ArrayRef GetByValArgRegs() const; /// The registers to use for the variable argument list. - ArrayRef GetVarArgRegs() const; + ArrayRef getVarArgRegs(bool isGP64bit) const; /// Obtain the size of the area allocated by the callee for arguments. /// CallingConv::FastCall affects the value for O32. diff --git a/llvm/lib/Target/Mips/MipsCallLowering.cpp b/llvm/lib/Target/Mips/MipsCallLowering.cpp index b856290211277..01c9d0b38323e 100644 --- a/llvm/lib/Target/Mips/MipsCallLowering.cpp +++ b/llvm/lib/Target/Mips/MipsCallLowering.cpp @@ -406,7 +406,8 @@ bool MipsCallLowering::lowerFormalArguments(MachineIRBuilder &MIRBuilder, return false; if (F.isVarArg()) { - ArrayRef ArgRegs = ABI.GetVarArgRegs(); + ArrayRef ArgRegs = + ABI.getVarArgRegs(MF.getSubtarget().isGP64bit()); unsigned Idx = CCInfo.getFirstUnallocated(ArgRegs); int VaArgOffset; diff --git a/llvm/lib/Target/Mips/MipsCallingConv.td b/llvm/lib/Target/Mips/MipsCallingConv.td index 25384a3fe8de3..3c60114f507b9 100644 --- a/llvm/lib/Target/Mips/MipsCallingConv.td +++ b/llvm/lib/Target/Mips/MipsCallingConv.td @@ -339,7 +339,7 @@ def CC_Mips_FixedArg : CallingConv<[ CCIfCC<"CallingConv::Fast", CCDelegateTo>, - CCIfSubtarget<"isABI_O32()", CCDelegateTo>, + CCIfSubtarget<"isABI_O32()", CCIfSubtargetNot<"isGP64bit()", CCDelegateTo>>, CCDelegateTo ]>; diff --git a/llvm/lib/Target/Mips/MipsISelLowering.cpp b/llvm/lib/Target/Mips/MipsISelLowering.cpp index fa8a4704730cf..55fc636d3c781 100644 --- a/llvm/lib/Target/Mips/MipsISelLowering.cpp +++ b/llvm/lib/Target/Mips/MipsISelLowering.cpp @@ -3400,7 +3400,6 @@ MipsTargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI, SDValue StackPtr = DAG.getCopyFromReg(Chain, DL, ABI.IsN64() ? Mips::SP_64 : Mips::SP, getPointerTy(DAG.getDataLayout())); - std::deque> RegsToPass; SmallVector MemOpChains; @@ -4654,7 +4653,7 @@ void MipsTargetLowering::writeVarArgRegs(std::vector &OutChains, SDValue Chain, const SDLoc &DL, SelectionDAG &DAG, CCState &State) const { - ArrayRef ArgRegs = ABI.GetVarArgRegs(); + ArrayRef ArgRegs = ABI.getVarArgRegs(Subtarget.isGP64bit()); unsigned Idx = State.getFirstUnallocated(ArgRegs); unsigned RegSizeInBytes = Subtarget.getGPRSizeInBytes(); MVT RegTy = MVT::getIntegerVT(RegSizeInBytes * 8); diff --git a/llvm/test/CodeGen/Mips/vararg.ll b/llvm/test/CodeGen/Mips/vararg.ll new file mode 100644 index 0000000000000..ed4a805af0c0c --- /dev/null +++ b/llvm/test/CodeGen/Mips/vararg.ll @@ -0,0 +1,54 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py +; RUN: llc -mtriple=mipsel-linux-gnu -mcpu=mips3 -target-abi o32 < %s | FileCheck %s -check-prefixes=MIPS3-O32 +; RUN: llc -mtriple=mipsel-linux-gnu -mcpu=mips3 -target-abi n32 < %s | FileCheck %s -check-prefixes=MIPS3-N32 +; RUN: llc -mtriple=mipsel-linux-gnu -mcpu=mips3 -target-abi n64 < %s | FileCheck %s -check-prefixes=MIPS3-N64 + +define void @func(ptr nocapture %x, ...) nounwind { +; MIPS3-O32-LABEL: func: +; MIPS32-O32: # %bb.0: # %entry +; MIPS32-O32-NEXT: addiu $sp, $sp, -48 +; MIPS32-O32-NEXT: sd $11, 56($sp) +; MIPS32-O32-NEXT: sd $10, 48($sp) +; MIPS32-O32-NEXT: sd $9, 40($sp) +; MIPS32-O32-NEXT: sd $8, 32($sp) +; MIPS32-O32-NEXT: sd $7, 24($sp) +; MIPS32-O32-NEXT: sd $6, 16($sp) +; MIPS32-O32-NEXT: sd $5, 8($sp) +; MIPS32-O32-NEXT: sw $4, 4($sp) +; MIPS32-O32-NEXT: jr $ra +; MIPS32-O32-NEXT: addiu $sp, $sp, 48 +; +; MIPS3-N32-LABEL: func: +; MIPS32-N32: # %bb.0: # %entry +; MIPS32-N32-NEXT: addiu $sp, $sp, -64 +; MIPS32-N32-NEXT: sd $11, 56($sp) +; MIPS32-N32-NEXT: sd $10, 48($sp) +; MIPS32-N32-NEXT: sd $9, 40($sp) +; MIPS32-N32-NEXT: sd $8, 32($sp) +; MIPS32-N32-NEXT: sd $7, 24($sp) +; MIPS32-N32-NEXT: sd $6, 16($sp) +; MIPS32-N32-NEXT: sd $5, 8($sp) +; MIPS32-N32-NEXT: sw $4, 4($sp) +; MIPS32-N32-NEXT: jr $ra +; MIPS32-N32-NEXT: addiu $sp, $sp, 64 +; +; MIPS3-N64-LABEL: func: +; MIPS32-N64: # %bb.0: # %entry +; MIPS32-N64-NEXT: addiu $sp, $sp, -64 +; MIPS32-N64-NEXT: sdl $4, 7($sp) +; MIPS32-N64-NEXT: sd $11, 56($sp) +; MIPS32-N64-NEXT: sd $10, 48($sp) +; MIPS32-N64-NEXT: sd $9, 40($sp) +; MIPS32-N64-NEXT: sd $8, 32($sp) +; MIPS32-N64-NEXT: sd $7, 24($sp) +; MIPS32-N64-NEXT: sd $6, 16($sp) +; MIPS32-N64-NEXT: sd $5, 8($sp) +; MIPS32-N64-NEXT: sw $4, 4($sp) +; MIPS32-N64-NEXT: jr $ra +; MIPS32-N64-NEXT: addiu $sp, $sp, 64 + +entry: + %x.addr = alloca ptr, align 4 + store ptr %x, ptr %x.addr, align 4 + ret void +} From 517605c20e6014543e91d45524a17c443aa11bd4 Mon Sep 17 00:00:00 2001 From: Ryosuke Niwa Date: Tue, 15 Apr 2025 20:00:51 -0700 Subject: [PATCH 58/63] [alpha.webkit.UnretainedCallArgsChecker] Add the support for RetainPtrArc (#135532) WebKit uses #define to rename RetainPtr to RetainPtrArc so add the support for it. --- .../Checkers/WebKit/PtrTypesSemantics.cpp | 15 +++++++++------ .../Checkers/WebKit/RetainPtrCtorAdoptChecker.cpp | 8 ++++---- .../Analysis/Checkers/WebKit/objc-mock-types.h | 5 +++++ .../Checkers/WebKit/unretained-call-args-arc.mm | 11 +++++++++++ .../Checkers/WebKit/unretained-call-args.mm | 11 +++++++++++ 5 files changed, 40 insertions(+), 10 deletions(-) diff --git a/clang/lib/StaticAnalyzer/Checkers/WebKit/PtrTypesSemantics.cpp b/clang/lib/StaticAnalyzer/Checkers/WebKit/PtrTypesSemantics.cpp index 134afcd124526..811888e119449 100644 --- a/clang/lib/StaticAnalyzer/Checkers/WebKit/PtrTypesSemantics.cpp +++ b/clang/lib/StaticAnalyzer/Checkers/WebKit/PtrTypesSemantics.cpp @@ -119,7 +119,9 @@ bool isRefType(const std::string &Name) { Name == "RefPtr" || Name == "RefPtrAllowingPartiallyDestroyed"; } -bool isRetainPtr(const std::string &Name) { return Name == "RetainPtr"; } +bool isRetainPtr(const std::string &Name) { + return Name == "RetainPtr" || Name == "RetainPtrArc"; +} bool isCheckedPtr(const std::string &Name) { return Name == "CheckedPtr" || Name == "CheckedRef"; @@ -157,7 +159,8 @@ bool isCtorOfCheckedPtr(const clang::FunctionDecl *F) { bool isCtorOfRetainPtr(const clang::FunctionDecl *F) { const std::string &FunctionName = safeGetName(F); return FunctionName == "RetainPtr" || FunctionName == "adoptNS" || - FunctionName == "adoptCF" || FunctionName == "retainPtr"; + FunctionName == "adoptCF" || FunctionName == "retainPtr" || + FunctionName == "RetainPtrArc" || FunctionName == "adoptNSArc"; } bool isCtorOfSafePtr(const clang::FunctionDecl *F) { @@ -190,7 +193,7 @@ bool isRefOrCheckedPtrType(const clang::QualType T) { } bool isRetainPtrType(const clang::QualType T) { - return isPtrOfType(T, [](auto Name) { return Name == "RetainPtr"; }); + return isPtrOfType(T, [](auto Name) { return isRetainPtr(Name); }); } bool isOwnerPtrType(const clang::QualType T) { @@ -374,7 +377,7 @@ std::optional isGetterOfSafePtr(const CXXMethodDecl *M) { method == "impl")) return true; - if (className == "RetainPtr" && method == "get") + if (isRetainPtr(className) && method == "get") return true; // Ref -> T conversion @@ -395,7 +398,7 @@ std::optional isGetterOfSafePtr(const CXXMethodDecl *M) { } } - if (className == "RetainPtr") { + if (isRetainPtr(className)) { if (auto *maybeRefToRawOperator = dyn_cast(M)) { auto QT = maybeRefToRawOperator->getConversionType(); auto *T = QT.getTypePtrOrNull(); @@ -429,7 +432,7 @@ bool isCheckedPtr(const CXXRecordDecl *R) { bool isRetainPtr(const CXXRecordDecl *R) { assert(R); if (auto *TmplR = R->getTemplateInstantiationPattern()) - return safeGetName(TmplR) == "RetainPtr"; + return isRetainPtr(safeGetName(TmplR)); return false; } diff --git a/clang/lib/StaticAnalyzer/Checkers/WebKit/RetainPtrCtorAdoptChecker.cpp b/clang/lib/StaticAnalyzer/Checkers/WebKit/RetainPtrCtorAdoptChecker.cpp index d372c5d1ba626..d3eee11311d91 100644 --- a/clang/lib/StaticAnalyzer/Checkers/WebKit/RetainPtrCtorAdoptChecker.cpp +++ b/clang/lib/StaticAnalyzer/Checkers/WebKit/RetainPtrCtorAdoptChecker.cpp @@ -71,7 +71,7 @@ class RetainPtrCtorAdoptChecker } bool TraverseClassTemplateDecl(ClassTemplateDecl *CTD) { - if (safeGetName(CTD) == "RetainPtr") + if (isRetainPtr(safeGetName(CTD))) return true; // Skip the contents of RetainPtr. return Base::TraverseClassTemplateDecl(CTD); } @@ -193,7 +193,7 @@ class RetainPtrCtorAdoptChecker if (!Cls) return; - if (safeGetName(Cls) != "RetainPtr" || !CE->getNumArgs()) + if (!isRetainPtr(safeGetName(Cls)) || !CE->getNumArgs()) return; // Ignore RetainPtr construction inside adoptNS, adoptCF, and retainPtr. @@ -322,12 +322,12 @@ class RetainPtrCtorAdoptChecker if (auto *CD = dyn_cast(MD)) { auto QT = CD->getConversionType().getCanonicalType(); auto *ResultType = QT.getTypePtrOrNull(); - if (safeGetName(Cls) == "RetainPtr" && ResultType && + if (isRetainPtr(safeGetName(Cls)) && ResultType && (ResultType->isPointerType() || ResultType->isReferenceType() || ResultType->isObjCObjectPointerType())) return IsOwnedResult::NotOwned; } - if (safeGetName(MD) == "leakRef" && safeGetName(Cls) == "RetainPtr") + if (safeGetName(MD) == "leakRef" && isRetainPtr(safeGetName(Cls))) return IsOwnedResult::Owned; } } diff --git a/clang/test/Analysis/Checkers/WebKit/objc-mock-types.h b/clang/test/Analysis/Checkers/WebKit/objc-mock-types.h index 51de81ac0f033..a4332df682060 100644 --- a/clang/test/Analysis/Checkers/WebKit/objc-mock-types.h +++ b/clang/test/Analysis/Checkers/WebKit/objc-mock-types.h @@ -17,6 +17,7 @@ typedef const struct CF_BRIDGED_TYPE(NSString) __CFString * CFStringRef; typedef const struct CF_BRIDGED_TYPE(NSArray) __CFArray * CFArrayRef; typedef struct CF_BRIDGED_MUTABLE_TYPE(NSMutableArray) __CFArray * CFMutableArrayRef; typedef struct CF_BRIDGED_MUTABLE_TYPE(CFRunLoopRef) __CFRunLoop * CFRunLoopRef; +typedef struct CF_BRIDGED_TYPE(id) CGImage *CGImageRef; #define NS_RETURNS_RETAINED __attribute__((ns_returns_retained)) #define CF_CONSUMED __attribute__((cf_consumed)) @@ -150,6 +151,10 @@ namespace WTF { void WTFCrash(void); +#if __has_feature(objc_arc) +#define RetainPtr RetainPtrArc +#endif + template class RetainPtr; template RetainPtr adoptNS(T*); template RetainPtr adoptCF(T); diff --git a/clang/test/Analysis/Checkers/WebKit/unretained-call-args-arc.mm b/clang/test/Analysis/Checkers/WebKit/unretained-call-args-arc.mm index f1f4d912663aa..4207c1836079f 100644 --- a/clang/test/Analysis/Checkers/WebKit/unretained-call-args-arc.mm +++ b/clang/test/Analysis/Checkers/WebKit/unretained-call-args-arc.mm @@ -5,6 +5,8 @@ SomeObj *provide(); CFMutableArrayRef provide_cf(); void someFunction(); +CGImageRef provideImage(); +NSString *stringForImage(CGImageRef); namespace raw_ptr { @@ -36,4 +38,13 @@ - (SomeObj *)getSomeObj { - (void)doWorkOnSomeObj { [[self getSomeObj] doWork]; } + +- (CGImageRef)createImage { + return provideImage(); +} + +- (NSString *)convertImage { + RetainPtr image = [self createImage]; + return stringForImage(image.get()); +} @end diff --git a/clang/test/Analysis/Checkers/WebKit/unretained-call-args.mm b/clang/test/Analysis/Checkers/WebKit/unretained-call-args.mm index 0667e4964f1a8..eb36b49313d42 100644 --- a/clang/test/Analysis/Checkers/WebKit/unretained-call-args.mm +++ b/clang/test/Analysis/Checkers/WebKit/unretained-call-args.mm @@ -9,6 +9,9 @@ CFMutableArrayRef provide_cf(); void consume_cf(CFMutableArrayRef); +CGImageRef provideImage(); +NSString *stringForImage(CGImageRef); + void some_function(); namespace simple { @@ -440,4 +443,12 @@ - (void)doWorkOnSomeObj { [[self getSomeObj] doWork]; } +- (CGImageRef)createImage { + return provideImage(); +} + +- (NSString *)convertImage { + RetainPtr image = [self createImage]; + return stringForImage(image.get()); +} @end From 71d091699f956c89135bc165165e815ab7876359 Mon Sep 17 00:00:00 2001 From: Alexey Samsonov Date: Tue, 15 Apr 2025 20:12:17 -0700 Subject: [PATCH 59/63] [libc][bazel] Fold "libc_function_deps" into "deps" for libc_tests. (#135835) libc_function_deps and deps are now identical, as we no longer need or have special treatment for libc_function targets. Merge these attributes passed to the libc_test macro, and fix all relevant libc_test macro invocations. This change is a no-op. This concludes cleanup started in 9b13d345303d819bb83de7ebbeb826d704add0bc. --- .../libc/test/libc_test_rules.bzl | 7 +- .../libc/test/src/complex/BUILD.bazel | 12 +-- .../libc/test/src/fenv/BUILD.bazel | 52 +++------ .../libc/test/src/inttypes/BUILD.bazel | 6 +- .../test/src/math/libc_math_test_rules.bzl | 3 +- .../libc/test/src/stdbit/BUILD.bazel | 11 +- .../libc/test/src/stdio/BUILD.bazel | 46 ++++---- .../libc/test/src/stdlib/BUILD.bazel | 100 +++++++++++------- .../libc/test/src/string/BUILD.bazel | 48 ++++----- .../libc/test/src/strings/BUILD.bazel | 12 +-- .../libc/test/src/sys/epoll/BUILD.bazel | 50 ++++----- .../libc/test/src/sys/socket/BUILD.bazel | 42 +++----- .../libc/test/src/unistd/BUILD.bazel | 100 +++++++----------- 13 files changed, 208 insertions(+), 281 deletions(-) diff --git a/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl b/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl index 7e798429ef19b..123e05727aeff 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl +++ b/utils/bazel/llvm-project-overlay/libc/test/libc_test_rules.bzl @@ -15,14 +15,13 @@ When performing tests we make sure to always use the internal version. load("//libc:libc_build_rules.bzl", "libc_common_copts") load("//libc:libc_configure_options.bzl", "LIBC_CONFIGURE_OPTIONS") -def libc_test(name, libc_function_deps = [], copts = [], deps = [], local_defines = [], **kwargs): +def libc_test(name, copts = [], deps = [], local_defines = [], **kwargs): """Add target for a libc test. Args: name: Test target name - libc_function_deps: List of libc_function targets used by this test. copts: The list of options to add to the C++ compilation command. - deps: The list of other libraries to be linked in to the test target. + deps: The list of libc functions and libraries to be linked in. local_defines: The list of target local_defines if any. **kwargs: Attributes relevant for a cc_test. """ @@ -37,7 +36,7 @@ def libc_test(name, libc_function_deps = [], copts = [], deps = [], local_define "//libc:func_free", "//libc:func_malloc", "//libc:func_realloc", - ] + libc_function_deps + deps, + ] + deps, copts = copts + libc_common_copts(), linkstatic = 1, **kwargs diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/complex/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/complex/BUILD.bazel index d0965bb2ee147..cc3a8d8b4b96a 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/complex/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/complex/BUILD.bazel @@ -7,8 +7,7 @@ load("//libc/test:libc_test_rules.bzl", "libc_test") "CImagTest.h", func_name + "_test.cpp", ], - libc_function_deps = ["//libc:func_name".replace("func_name", func_name)], - deps = [ + deps = ["//libc:func_name".replace("func_name", func_name)] + [ "//libc:hdr_math_macros", "//libc/test/UnitTest:fp_test_helpers", ], @@ -29,8 +28,7 @@ load("//libc/test:libc_test_rules.bzl", "libc_test") "ConjTest.h", func_name + "_test.cpp", ], - libc_function_deps = ["//libc:func_name".replace("func_name", func_name)], - deps = [ + deps = ["//libc:func_name".replace("func_name", func_name)] + [ "//libc:hdr_math_macros", "//libc/test/UnitTest:fp_test_helpers", ], @@ -51,8 +49,7 @@ load("//libc/test:libc_test_rules.bzl", "libc_test") "CprojTest.h", func_name + "_test.cpp", ], - libc_function_deps = ["//libc:func_name".replace("func_name", func_name)], - deps = [ + deps = ["//libc:func_name".replace("func_name", func_name)] + [ "//libc:hdr_math_macros", "//libc/test/UnitTest:fp_test_helpers", ] + (["//libc/utils/MPCWrapper:mpc_wrapper"] if func_name == "cprojf" else []), @@ -73,8 +70,7 @@ load("//libc/test:libc_test_rules.bzl", "libc_test") "CRealTest.h", func_name + "_test.cpp", ], - libc_function_deps = ["//libc:func_name".replace("func_name", func_name)], - deps = [ + deps = ["//libc:func_name".replace("func_name", func_name)] + [ "//libc:hdr_math_macros", "//libc/test/UnitTest:fp_test_helpers", ], diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/fenv/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/fenv/BUILD.bazel index c6ae534b0f640..1af1a984db92a 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/fenv/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/fenv/BUILD.bazel @@ -16,14 +16,12 @@ libc_test( "exception_status_test.cpp", "excepts.h", ], - libc_function_deps = [ + deps = [ + "//libc:__support_fputil_fenv_impl", "//libc:feclearexcept", "//libc:feraiseexcept", "//libc:fesetexcept", "//libc:fetestexcept", - ], - deps = [ - "//libc:__support_fputil_fenv_impl", "//libc:hdr_fenv_macros", "//libc/test/UnitTest:fp_test_helpers", ], @@ -35,11 +33,9 @@ libc_test( "excepts.h", "rounding_mode_test.cpp", ], - libc_function_deps = [ + deps = [ "//libc:fegetround", "//libc:fesetround", - ], - deps = [ "//libc:hdr_fenv_macros", "//libc/test/UnitTest:fp_test_helpers", ], @@ -51,16 +47,14 @@ libc_test( "enabled_exceptions_test.cpp", "excepts.h", ], - libc_function_deps = [ - "//libc:feclearexcept", - "//libc:feraiseexcept", - "//libc:fetestexcept", - ], tags = ["nosan"], deps = [ "//libc:__support_common", "//libc:__support_fputil_fenv_impl", "//libc:__support_macros_properties_architectures", + "//libc:feclearexcept", + "//libc:feraiseexcept", + "//libc:fetestexcept", "//libc:hdr_fenv_macros", "//libc/test/UnitTest:fp_test_helpers", ], @@ -72,14 +66,12 @@ libc_test( "excepts.h", "feholdexcept_test.cpp", ], - libc_function_deps = [ - "//libc:feholdexcept", - ], tags = ["nosan"], deps = [ "//libc:__support_common", "//libc:__support_fputil_fenv_impl", "//libc:__support_macros_properties_architectures", + "//libc:feholdexcept", "//libc:hdr_fenv_macros", "//libc:types_fenv_t", "//libc/test/UnitTest:fp_test_helpers", @@ -92,13 +84,11 @@ libc_test( "exception_flags_test.cpp", "excepts.h", ], - libc_function_deps = [ + deps = [ + "//libc:__support_fputil_fenv_impl", "//libc:fegetexceptflag", "//libc:fesetexceptflag", "//libc:fetestexceptflag", - ], - deps = [ - "//libc:__support_fputil_fenv_impl", "//libc:hdr_fenv_macros", "//libc:types_fexcept_t", "//libc/test/UnitTest:fp_test_helpers", @@ -111,11 +101,9 @@ libc_test( "excepts.h", "feclearexcept_test.cpp", ], - libc_function_deps = [ - "//libc:feclearexcept", - ], deps = [ "//libc:__support_fputil_fenv_impl", + "//libc:feclearexcept", "//libc:hdr_fenv_macros", "//libc/test/UnitTest:fp_test_helpers", ], @@ -127,14 +115,12 @@ libc_test( "excepts.h", "feenableexcept_test.cpp", ], - libc_function_deps = [ - "//libc:fedisableexcept", - "//libc:feenableexcept", - "//libc:fegetexcept", - ], deps = [ "//libc:__support_common", "//libc:__support_macros_properties_architectures", + "//libc:fedisableexcept", + "//libc:feenableexcept", + "//libc:fegetexcept", "//libc:hdr_fenv_macros", "//libc/test/UnitTest:fp_test_helpers", ], @@ -146,11 +132,9 @@ libc_test( "excepts.h", "feupdateenv_test.cpp", ], - libc_function_deps = [ - "//libc:feupdateenv", - ], deps = [ "//libc:__support_fputil_fenv_impl", + "//libc:feupdateenv", "//libc:hdr_fenv_macros", "//libc:types_fenv_t", "//libc/test/UnitTest:fp_test_helpers", @@ -163,15 +147,13 @@ libc_test( "excepts.h", "getenv_and_setenv_test.cpp", ], - libc_function_deps = [ + deps = [ + "//libc:__support_fputil_fenv_impl", + "//libc:__support_macros_properties_os", "//libc:fegetenv", "//libc:fegetround", "//libc:fesetenv", "//libc:fesetround", - ], - deps = [ - "//libc:__support_fputil_fenv_impl", - "//libc:__support_macros_properties_os", "//libc:hdr_fenv_macros", "//libc:types_fenv_t", "//libc/test/UnitTest:fp_test_helpers", diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/inttypes/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/inttypes/BUILD.bazel index bda7245d1f677..3dd4ab379efe0 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/inttypes/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/inttypes/BUILD.bazel @@ -13,7 +13,7 @@ licenses(["notice"]) libc_test( name = "imaxabs_test", srcs = ["imaxabs_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:imaxabs", ], ) @@ -21,10 +21,8 @@ libc_test( libc_test( name = "imaxdiv_test", srcs = ["imaxdiv_test.cpp"], - libc_function_deps = [ - "//libc:imaxdiv", - ], deps = [ + "//libc:imaxdiv", "//libc/test/src/stdlib:div_test_helper", ], ) diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl b/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl index 16845ab66dfd4..d2297d51383cc 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl +++ b/utils/bazel/llvm-project-overlay/libc/test/src/math/libc_math_test_rules.bzl @@ -23,8 +23,7 @@ def math_test(name, hdrs = [], deps = [], **kwargs): libc_test( name = test_name, srcs = [test_name + ".cpp"] + hdrs, - libc_function_deps = ["//libc:func_name".replace("func_name", name)], - deps = [ + deps = ["//libc:func_name".replace("func_name", name)] + [ "//libc:__support_cpp_algorithm", "//libc:__support_cpp_bit", "//libc:__support_cpp_limits", diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/stdbit/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/stdbit/BUILD.bazel index b9d153947dc7d..119d7d5337750 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/stdbit/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/stdbit/BUILD.bazel @@ -39,11 +39,12 @@ bit_prefix_list = [ libc_test( name = bit_prefix + bit_suffix + "_test", srcs = [bit_prefix + bit_suffix + "_test.cpp"], - libc_function_deps = ["//libc:func_name".replace( - "func_name", - bit_prefix + bit_suffix, - )], - deps = ["//libc:__support_cpp_limits"], + deps = [ + "//libc:func_name".replace( + "func_name", + bit_prefix + bit_suffix, + ), + ] + ["//libc:__support_cpp_limits"], ) for bit_prefix in bit_prefix_list for bit_suffix in bit_suffix_list diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/stdio/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/stdio/BUILD.bazel index c3865ea07ea91..484d3e5e0a24e 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/stdio/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/stdio/BUILD.bazel @@ -13,8 +13,6 @@ licenses(["notice"]) libc_test( name = "printf_parser_test", srcs = ["printf_core/parser_test.cpp"], - libc_function_deps = [ - ], deps = [ "//libc:__support_arg_list", "//libc:__support_cpp_bit", @@ -28,8 +26,6 @@ libc_test( libc_test( name = "printf_writer_test", srcs = ["printf_core/writer_test.cpp"], - libc_function_deps = [ - ], deps = [ "//libc:__support_arg_list", "//libc:__support_cpp_string_view", @@ -42,8 +38,6 @@ libc_test( libc_test( name = "printf_converter_test", srcs = ["printf_core/converter_test.cpp"], - libc_function_deps = [ - ], deps = [ "//libc:__support_arg_list", "//libc:__support_cpp_string_view", @@ -56,11 +50,9 @@ libc_test( libc_test( name = "sprintf_test", srcs = ["sprintf_test.cpp"], - libc_function_deps = [ - "//libc:sprintf", - ], deps = [ "//libc:__support_fputil_fp_bits", + "//libc:sprintf", "//libc/test/UnitTest:fp_test_helpers", ], ) @@ -68,7 +60,7 @@ libc_test( libc_test( name = "snprintf_test", srcs = ["snprintf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:snprintf", ], ) @@ -76,7 +68,7 @@ libc_test( libc_test( name = "printf_test", srcs = ["printf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:printf", ], ) @@ -84,7 +76,7 @@ libc_test( libc_test( name = "fprintf_test", srcs = ["fprintf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:fprintf", ], ) @@ -92,7 +84,7 @@ libc_test( libc_test( name = "vsprintf_test", srcs = ["vsprintf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:vsprintf", ], ) @@ -100,7 +92,7 @@ libc_test( libc_test( name = "vsnprintf_test", srcs = ["vsnprintf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:vsnprintf", ], ) @@ -108,7 +100,7 @@ libc_test( libc_test( name = "vprintf_test", srcs = ["vprintf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:vprintf", ], ) @@ -116,7 +108,7 @@ libc_test( libc_test( name = "vfprintf_test", srcs = ["vfprintf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:vfprintf", ], ) @@ -124,25 +116,23 @@ libc_test( libc_test( name = "remove_test", srcs = ["remove_test.cpp"], - libc_function_deps = [ - "//libc:remove", - "//libc:open", - "//libc:mkdirat", + deps = [ "//libc:access", "//libc:close", + "//libc:mkdirat", + "//libc:open", + "//libc:remove", ], ) libc_test( name = "sscanf_test", srcs = ["sscanf_test.cpp"], - libc_function_deps = [ - "//libc:sscanf", - ], deps = [ "//libc:__support_cpp_limits", "//libc:__support_fputil_fp_bits", "//libc:hdr_stdio_macros", + "//libc:sscanf", "//libc/test/UnitTest:fp_test_helpers", ], ) @@ -150,16 +140,16 @@ libc_test( libc_test( name = "fscanf_test", srcs = ["fscanf_test.cpp"], - libc_function_deps = [ + deps = [ + "//libc:__support_cpp_string_view", "//libc:fscanf", ], - deps = ["//libc:__support_cpp_string_view"], ) libc_test( name = "vsscanf_test", srcs = ["vsscanf_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:vsscanf", ], ) @@ -167,8 +157,8 @@ libc_test( libc_test( name = "vfscanf_test", srcs = ["vfscanf_test.cpp"], - libc_function_deps = [ + deps = [ + "//libc:__support_cpp_string_view", "//libc:vfscanf", ], - deps = ["//libc:__support_cpp_string_view"], ) diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel index 5f43ec7c7a109..40f672d8099f1 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/stdlib/BUILD.bazel @@ -13,19 +13,19 @@ licenses(["notice"]) libc_test( name = "abs_test", srcs = ["abs_test.cpp"], - libc_function_deps = ["//libc:abs"], + deps = ["//libc:abs"], ) libc_test( name = "labs_test", srcs = ["labs_test.cpp"], - libc_function_deps = ["//libc:labs"], + deps = ["//libc:labs"], ) libc_test( name = "llabs_test", srcs = ["llabs_test.cpp"], - libc_function_deps = ["//libc:llabs"], + deps = ["//libc:llabs"], ) libc_test_library( @@ -37,9 +37,9 @@ libc_test_library( libc_test( name = "div_test", srcs = ["div_test.cpp"], - libc_function_deps = ["//libc:div"], deps = [ ":div_test_helper", + "//libc:div", "//libc:types_div_t", ], ) @@ -47,9 +47,9 @@ libc_test( libc_test( name = "ldiv_test", srcs = ["ldiv_test.cpp"], - libc_function_deps = ["//libc:ldiv"], deps = [ ":div_test_helper", + "//libc:ldiv", "//libc:types_ldiv_t", ], ) @@ -57,9 +57,9 @@ libc_test( libc_test( name = "lldiv_test", srcs = ["lldiv_test.cpp"], - libc_function_deps = ["//libc:lldiv"], deps = [ ":div_test_helper", + "//libc:lldiv", "//libc:types_lldiv_t", ], ) @@ -77,36 +77,46 @@ libc_test_library( libc_test( name = "atoi_test", srcs = ["atoi_test.cpp"], - libc_function_deps = ["//libc:atoi"], - deps = [":atoi_test_helper"], + deps = [ + ":atoi_test_helper", + "//libc:atoi", + ], ) libc_test( name = "atol_test", srcs = ["atol_test.cpp"], - libc_function_deps = ["//libc:atol"], - deps = [":atoi_test_helper"], + deps = [ + ":atoi_test_helper", + "//libc:atol", + ], ) libc_test( name = "atoll_test", srcs = ["atoll_test.cpp"], - libc_function_deps = ["//libc:atoll"], - deps = [":atoi_test_helper"], + deps = [ + ":atoi_test_helper", + "//libc:atoll", + ], ) libc_test( name = "atof_test", srcs = ["atof_test.cpp"], - libc_function_deps = ["//libc:atof"], - deps = ["//libc:__support_fputil_fp_bits"], + deps = [ + "//libc:__support_fputil_fp_bits", + "//libc:atof", + ], ) libc_test( name = "bsearch_test", srcs = ["bsearch_test.cpp"], - libc_function_deps = ["//libc:bsearch"], - deps = ["//libc:types_size_t"], + deps = [ + "//libc:bsearch", + "//libc:types_size_t", + ], ) libc_test_library( @@ -121,9 +131,9 @@ libc_test_library( libc_test( name = "quick_sort_test", srcs = ["quick_sort_test.cpp"], - libc_function_deps = ["//libc:qsort"], deps = [ ":qsort_test_helper", + "//libc:qsort", "//libc:qsort_util", "//libc:types_size_t", ], @@ -132,9 +142,9 @@ libc_test( libc_test( name = "heap_sort_test", srcs = ["heap_sort_test.cpp"], - libc_function_deps = ["//libc:qsort"], deps = [ ":qsort_test_helper", + "//libc:qsort", "//libc:qsort_util", "//libc:types_size_t", ], @@ -143,8 +153,10 @@ libc_test( libc_test( name = "qsort_r_test", srcs = ["qsort_r_test.cpp"], - libc_function_deps = ["//libc:qsort_r"], - deps = ["//libc:types_size_t"], + deps = [ + "//libc:qsort_r", + "//libc:types_size_t", + ], ) libc_test_library( @@ -160,22 +172,28 @@ libc_test_library( libc_test( name = "strfromf_test", srcs = ["strfromf_test.cpp"], - libc_function_deps = ["//libc:strfromf"], - deps = [":strfrom_test_helper"], + deps = [ + ":strfrom_test_helper", + "//libc:strfromf", + ], ) libc_test( name = "strfromd_test", srcs = ["strfromd_test.cpp"], - libc_function_deps = ["//libc:strfromd"], - deps = [":strfrom_test_helper"], + deps = [ + ":strfrom_test_helper", + "//libc:strfromd", + ], ) libc_test( name = "strfroml_test", srcs = ["strfroml_test.cpp"], - libc_function_deps = ["//libc:strfroml"], - deps = [":strfrom_test_helper"], + deps = [ + ":strfrom_test_helper", + "//libc:strfroml", + ], ) libc_test_library( @@ -194,37 +212,45 @@ libc_test_library( libc_test( name = "strtol_test", srcs = ["strtol_test.cpp"], - libc_function_deps = ["//libc:strtol"], - deps = [":strtol_test_helper"], + deps = [ + ":strtol_test_helper", + "//libc:strtol", + ], ) libc_test( name = "strtoll_test", srcs = ["strtoll_test.cpp"], - libc_function_deps = ["//libc:strtoll"], - deps = [":strtol_test_helper"], + deps = [ + ":strtol_test_helper", + "//libc:strtoll", + ], ) libc_test( name = "strtoul_test", srcs = ["strtoul_test.cpp"], - libc_function_deps = ["//libc:strtoul"], - deps = [":strtol_test_helper"], + deps = [ + ":strtol_test_helper", + "//libc:strtoul", + ], ) libc_test( name = "strtoull_test", srcs = ["strtoull_test.cpp"], - libc_function_deps = ["//libc:strtoull"], - deps = [":strtol_test_helper"], + deps = [ + ":strtol_test_helper", + "//libc:strtoull", + ], ) libc_test( name = "strtof_test", srcs = ["strtof_test.cpp"], - libc_function_deps = ["//libc:strtof"], deps = [ "//libc:__support_fputil_fp_bits", + "//libc:strtof", "//libc/test/UnitTest:fp_test_helpers", ], ) @@ -232,9 +258,9 @@ libc_test( libc_test( name = "strtod_test", srcs = ["strtod_test.cpp"], - libc_function_deps = ["//libc:strtod"], deps = [ "//libc:__support_fputil_fp_bits", + "//libc:strtod", "//libc/test/UnitTest:fp_test_helpers", ], ) @@ -242,9 +268,9 @@ libc_test( libc_test( name = "strtold_test", srcs = ["strtold_test.cpp"], - libc_function_deps = ["//libc:strtold"], deps = [ "//libc:__support_fputil_fp_bits", "//libc:__support_uint128", + "//libc:strtold", ], ) diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/string/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/string/BUILD.bazel index 7274819e6758c..7555c9f35e3a0 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/string/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/string/BUILD.bazel @@ -13,7 +13,7 @@ licenses(["notice"]) libc_test( name = "strlen_test", srcs = ["strlen_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strlen", ], ) @@ -21,7 +21,7 @@ libc_test( libc_test( name = "strcpy_test", srcs = ["strcpy_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strcpy", ], ) @@ -29,7 +29,7 @@ libc_test( libc_test( name = "strcmp_test", srcs = ["strcmp_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strcmp", ], ) @@ -37,7 +37,7 @@ libc_test( libc_test( name = "memchr_test", srcs = ["memchr_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:memchr", ], ) @@ -51,16 +51,16 @@ libc_test_library( libc_test( name = "strchr_test", srcs = ["strchr_test.cpp"], - libc_function_deps = [ + deps = [ + ":strchr_test_helper", "//libc:strchr", ], - deps = [":strchr_test_helper"], ) libc_test( name = "strstr_test", srcs = ["strstr_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strstr", ], ) @@ -68,7 +68,7 @@ libc_test( libc_test( name = "strnlen_test", srcs = ["strnlen_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strnlen", ], ) @@ -76,7 +76,7 @@ libc_test( libc_test( name = "memrchr_test", srcs = ["memrchr_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:memrchr", ], ) @@ -84,16 +84,16 @@ libc_test( libc_test( name = "strrchr_test", srcs = ["strrchr_test.cpp"], - libc_function_deps = [ + deps = [ + ":strchr_test_helper", "//libc:strrchr", ], - deps = [":strchr_test_helper"], ) libc_test( name = "strcspn_test", srcs = ["strcspn_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strcspn", ], ) @@ -101,7 +101,7 @@ libc_test( libc_test( name = "strspn_test", srcs = ["strspn_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strspn", ], ) @@ -109,7 +109,7 @@ libc_test( libc_test( name = "strtok_test", srcs = ["strtok_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:strtok", ], ) @@ -138,20 +138,18 @@ libc_test_library( libc_test( name = "memcpy_test", srcs = ["memcpy_test.cpp"], - libc_function_deps = [ - "//libc:memcpy", - ], deps = [ ":memory_check_utils", ":protected_pages", "//libc:__support_macros_properties_os", + "//libc:memcpy", ], ) libc_test( name = "mempcpy_test", srcs = ["mempcpy_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:mempcpy", ], ) @@ -159,26 +157,22 @@ libc_test( libc_test( name = "memset_test", srcs = ["memset_test.cpp"], - libc_function_deps = [ - "//libc:memset", - ], deps = [ ":memory_check_utils", ":protected_pages", "//libc:__support_macros_properties_os", + "//libc:memset", ], ) libc_test( name = "memmove_test", srcs = ["memmove_test.cpp"], - libc_function_deps = [ - "//libc:memcmp", - "//libc:memmove", - ], deps = [ ":memory_check_utils", "//libc:__support_cpp_span", + "//libc:memcmp", + "//libc:memmove", "//libc/test/UnitTest:memory_matcher", ], ) @@ -186,11 +180,9 @@ libc_test( libc_test( name = "memcmp_test", srcs = ["memcmp_test.cpp"], - libc_function_deps = [ - "//libc:memcmp", - ], deps = [ ":memory_check_utils", + "//libc:memcmp", "//libc/test/UnitTest:test_logger", ], ) diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/strings/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/strings/BUILD.bazel index bcf4b4201e043..2e6f5644eec71 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/strings/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/strings/BUILD.bazel @@ -13,11 +13,9 @@ licenses(["notice"]) libc_test( name = "bcopy_test", srcs = ["bcopy_test.cpp"], - libc_function_deps = [ - "//libc:bcopy", - ], deps = [ "//libc:__support_cpp_span", + "//libc:bcopy", "//libc/test/UnitTest:memory_matcher", "//libc/test/src/string:memory_check_utils", ], @@ -26,10 +24,8 @@ libc_test( libc_test( name = "bcmp_test", srcs = ["bcmp_test.cpp"], - libc_function_deps = [ - "//libc:bcmp", - ], deps = [ + "//libc:bcmp", "//libc/test/UnitTest:test_logger", "//libc/test/src/string:memory_check_utils", ], @@ -38,10 +34,8 @@ libc_test( libc_test( name = "bzero_test", srcs = ["bzero_test.cpp"], - libc_function_deps = [ - "//libc:bzero", - ], deps = [ + "//libc:bzero", "//libc/test/src/string:memory_check_utils", ], ) diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/sys/epoll/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/sys/epoll/BUILD.bazel index 63ddebdadbdc9..e391703075aa7 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/sys/epoll/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/sys/epoll/BUILD.bazel @@ -13,11 +13,9 @@ licenses(["notice"]) libc_test( name = "epoll_create_test", srcs = ["linux/epoll_create_test.cpp"], - libc_function_deps = [ - "//libc:epoll_create", - "//libc:close", - ], deps = [ + "//libc:close", + "//libc:epoll_create", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -25,11 +23,9 @@ libc_test( libc_test( name = "epoll_create1_test", srcs = ["linux/epoll_create1_test.cpp"], - libc_function_deps = [ - "//libc:epoll_create1", - "//libc:close", - ], deps = [ + "//libc:close", + "//libc:epoll_create1", "//libc:hdr_sys_epoll_macros", "//libc/test/UnitTest:errno_test_helpers", ], @@ -38,14 +34,12 @@ libc_test( libc_test( name = "epoll_ctl_test", srcs = ["linux/epoll_ctl_test.cpp"], - libc_function_deps = [ + deps = [ + "//libc:close", "//libc:epoll_create1", "//libc:epoll_ctl", - "//libc:pipe", - "//libc:close", - ], - deps = [ "//libc:hdr_sys_epoll_macros", + "//libc:pipe", "//libc:types_struct_epoll_event", "//libc/test/UnitTest:errno_test_helpers", ], @@ -54,15 +48,13 @@ libc_test( libc_test( name = "epoll_wait_test", srcs = ["linux/epoll_wait_test.cpp"], - libc_function_deps = [ - "//libc:epoll_wait", + deps = [ + "//libc:close", "//libc:epoll_create1", "//libc:epoll_ctl", - "//libc:pipe", - "//libc:close", - ], - deps = [ + "//libc:epoll_wait", "//libc:hdr_sys_epoll_macros", + "//libc:pipe", "//libc:types_struct_epoll_event", "//libc/test/UnitTest:errno_test_helpers", ], @@ -71,15 +63,13 @@ libc_test( libc_test( name = "epoll_pwait_test", srcs = ["linux/epoll_pwait_test.cpp"], - libc_function_deps = [ - "//libc:epoll_pwait", + deps = [ + "//libc:close", "//libc:epoll_create1", "//libc:epoll_ctl", - "//libc:pipe", - "//libc:close", - ], - deps = [ + "//libc:epoll_pwait", "//libc:hdr_sys_epoll_macros", + "//libc:pipe", "//libc:types_struct_epoll_event", "//libc/test/UnitTest:errno_test_helpers", ], @@ -88,15 +78,13 @@ libc_test( libc_test( name = "epoll_pwait2_test", srcs = ["linux/epoll_pwait2_test.cpp"], - libc_function_deps = [ - "//libc:epoll_pwait2", + deps = [ + "//libc:close", "//libc:epoll_create1", "//libc:epoll_ctl", - "//libc:pipe", - "//libc:close", - ], - deps = [ + "//libc:epoll_pwait2", "//libc:hdr_sys_epoll_macros", + "//libc:pipe", "//libc:types_struct_epoll_event", "//libc:types_struct_timespec", "//libc/test/UnitTest:errno_test_helpers", diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/sys/socket/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/sys/socket/BUILD.bazel index ac7f48d0aeebb..743bf2a4743b7 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/sys/socket/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/sys/socket/BUILD.bazel @@ -13,11 +13,9 @@ licenses(["notice"]) libc_test( name = "socket_test", srcs = ["linux/socket_test.cpp"], - libc_function_deps = [ - "//libc:socket", - "//libc:close", - ], deps = [ + "//libc:close", + "//libc:socket", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -25,11 +23,9 @@ libc_test( libc_test( name = "socketpair_test", srcs = ["linux/socketpair_test.cpp"], - libc_function_deps = [ - "//libc:socketpair", - "//libc:close", - ], deps = [ + "//libc:close", + "//libc:socketpair", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -37,13 +33,11 @@ libc_test( libc_test( name = "send_recv_test", srcs = ["linux/send_recv_test.cpp"], - libc_function_deps = [ - "//libc:socketpair", - "//libc:send", - "//libc:recv", - "//libc:close", - ], deps = [ + "//libc:close", + "//libc:recv", + "//libc:send", + "//libc:socketpair", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -51,13 +45,11 @@ libc_test( libc_test( name = "sendto_recvfrom_test", srcs = ["linux/sendto_recvfrom_test.cpp"], - libc_function_deps = [ - "//libc:socketpair", - "//libc:sendto", - "//libc:recvfrom", - "//libc:close", - ], deps = [ + "//libc:close", + "//libc:recvfrom", + "//libc:sendto", + "//libc:socketpair", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -65,13 +57,11 @@ libc_test( libc_test( name = "sendmsg_recvmsg_test", srcs = ["linux/sendmsg_recvmsg_test.cpp"], - libc_function_deps = [ - "//libc:socketpair", - "//libc:sendmsg", - "//libc:recvmsg", - "//libc:close", - ], deps = [ + "//libc:close", + "//libc:recvmsg", + "//libc:sendmsg", + "//libc:socketpair", "//libc/test/UnitTest:errno_test_helpers", ], ) diff --git a/utils/bazel/llvm-project-overlay/libc/test/src/unistd/BUILD.bazel b/utils/bazel/llvm-project-overlay/libc/test/src/unistd/BUILD.bazel index 62641889f9a72..661e0a6ff5dfe 100644 --- a/utils/bazel/llvm-project-overlay/libc/test/src/unistd/BUILD.bazel +++ b/utils/bazel/llvm-project-overlay/libc/test/src/unistd/BUILD.bazel @@ -13,13 +13,11 @@ licenses(["notice"]) libc_test( name = "access_test", srcs = ["access_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:access", "//libc:close", + "//libc:open", "//libc:unlink", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -27,15 +25,13 @@ libc_test( libc_test( name = "dup_test", srcs = ["dup_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", "//libc:dup", + "//libc:open", "//libc:read", "//libc:unlink", "//libc:write", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -43,15 +39,13 @@ libc_test( libc_test( name = "dup2_test", srcs = ["dup2_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", "//libc:dup2", + "//libc:open", "//libc:read", "//libc:unlink", "//libc:write", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -59,15 +53,13 @@ libc_test( libc_test( name = "dup3_test", srcs = ["dup3_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", "//libc:dup3", + "//libc:open", "//libc:read", "//libc:unlink", "//libc:write", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -75,16 +67,14 @@ libc_test( libc_test( name = "ftruncate_test", srcs = ["ftruncate_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ + "//libc:__support_cpp_string_view", "//libc:close", - "//libc:read", "//libc:ftruncate", + "//libc:open", + "//libc:read", "//libc:unlink", "//libc:write", - ], - deps = [ - "//libc:__support_cpp_string_view", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -92,16 +82,14 @@ libc_test( libc_test( name = "pread_pwrite_test", srcs = ["pread_pwrite_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", "//libc:fsync", + "//libc:open", "//libc:pread", "//libc:pwrite", "//libc:unlink", "//libc:write", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -109,15 +97,13 @@ libc_test( libc_test( name = "read_write_test", srcs = ["read_write_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", "//libc:fsync", + "//libc:open", "//libc:read", - "//libc:write", "//libc:remove", - ], - deps = [ + "//libc:write", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -125,13 +111,11 @@ libc_test( libc_test( name = "link_test", srcs = ["link_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", "//libc:link", + "//libc:open", "//libc:unlink", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -139,24 +123,20 @@ libc_test( libc_test( name = "swab_test", srcs = ["swab_test.cpp"], - libc_function_deps = [ - "//libc:swab", - ], deps = [ "//libc:string_utils", + "//libc:swab", ], ) libc_test( name = "symlink_test", srcs = ["symlink_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", + "//libc:open", "//libc:symlink", "//libc:unlink", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -164,16 +144,14 @@ libc_test( libc_test( name = "truncate_test", srcs = ["truncate_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ + "//libc:__support_cpp_string_view", "//libc:close", + "//libc:open", "//libc:read", "//libc:truncate", "//libc:unlink", "//libc:write", - ], - deps = [ - "//libc:__support_cpp_string_view", "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -181,12 +159,10 @@ libc_test( libc_test( name = "unlink_test", srcs = ["unlink_test.cpp"], - libc_function_deps = [ - "//libc:open", + deps = [ "//libc:close", + "//libc:open", "//libc:unlink", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -194,7 +170,7 @@ libc_test( libc_test( name = "getppid_test", srcs = ["getppid_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:getppid", ], ) @@ -202,7 +178,7 @@ libc_test( libc_test( name = "getuid_test", srcs = ["getuid_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:getuid", ], ) @@ -210,12 +186,10 @@ libc_test( libc_test( name = "isatty_test", srcs = ["isatty_test.cpp"], - libc_function_deps = [ + deps = [ + "//libc:close", "//libc:isatty", "//libc:open", - "//libc:close", - ], - deps = [ "//libc/test/UnitTest:errno_test_helpers", ], ) @@ -223,7 +197,7 @@ libc_test( libc_test( name = "geteuid_test", srcs = ["geteuid_test.cpp"], - libc_function_deps = [ + deps = [ "//libc:geteuid", ], ) @@ -236,7 +210,7 @@ libc_test( # libc_test( # name = "syscall_test", # srcs = ["syscall_test.cpp"], -# libc_function_deps = [ +# deps = [ # "//libc:syscall", # ], # ) @@ -246,7 +220,7 @@ libc_test( # libc_test( # name = "sysconf_test", # srcs = ["sysconf_test.cpp"], -# libc_function_deps = [ +# deps = [ # "//libc:sysconf", # ], # ) @@ -256,12 +230,10 @@ libc_test( # libc_test( # name = "getopt_test", # srcs = ["getopt_test.cpp"], -# libc_function_deps = [ +# deps = [ +# "//libc:__support_cpp_array", # "//libc:getopt", # "//libc:fopencookie", # "//libc:fflush", # ], -# deps = [ -# "//libc:__support_cpp_array", -# ], # ) From a024d13f84dbe7b3d1eee555ddc82cdd1af814e0 Mon Sep 17 00:00:00 2001 From: tangaac Date: Wed, 16 Apr 2025 11:33:29 +0800 Subject: [PATCH 60/63] [LoongArch] make ABDS/ABDU legal for lsx/lasx (#134190) --- .../LoongArch/LoongArchISelLowering.cpp | 4 + .../LoongArch/LoongArchLASXInstrInfo.td | 4 + .../Target/LoongArch/LoongArchLSXInstrInfo.td | 4 + .../LoongArch/lasx/ir-instruction/absd.ll | 151 +++++------------- .../LoongArch/lsx/ir-instruction/absd.ll | 151 +++++------------- 5 files changed, 86 insertions(+), 228 deletions(-) diff --git a/llvm/lib/Target/LoongArch/LoongArchISelLowering.cpp b/llvm/lib/Target/LoongArch/LoongArchISelLowering.cpp index 002d88cbeeba3..e5ccbe897d19c 100644 --- a/llvm/lib/Target/LoongArch/LoongArchISelLowering.cpp +++ b/llvm/lib/Target/LoongArch/LoongArchISelLowering.cpp @@ -272,6 +272,8 @@ LoongArchTargetLowering::LoongArchTargetLowering(const TargetMachine &TM, {ISD::SETNE, ISD::SETGE, ISD::SETGT, ISD::SETUGE, ISD::SETUGT}, VT, Expand); setOperationAction(ISD::SCALAR_TO_VECTOR, VT, Custom); + setOperationAction(ISD::ABDS, VT, Legal); + setOperationAction(ISD::ABDU, VT, Legal); } for (MVT VT : {MVT::v16i8, MVT::v8i16, MVT::v4i32}) setOperationAction(ISD::BITREVERSE, VT, Custom); @@ -336,6 +338,8 @@ LoongArchTargetLowering::LoongArchTargetLowering(const TargetMachine &TM, {ISD::SETNE, ISD::SETGE, ISD::SETGT, ISD::SETUGE, ISD::SETUGT}, VT, Expand); setOperationAction(ISD::SCALAR_TO_VECTOR, VT, Custom); + setOperationAction(ISD::ABDS, VT, Legal); + setOperationAction(ISD::ABDU, VT, Legal); } for (MVT VT : {MVT::v32i8, MVT::v16i16, MVT::v8i32}) setOperationAction(ISD::BITREVERSE, VT, Custom); diff --git a/llvm/lib/Target/LoongArch/LoongArchLASXInstrInfo.td b/llvm/lib/Target/LoongArch/LoongArchLASXInstrInfo.td index d6d532cddb594..aaa4b94b6e994 100644 --- a/llvm/lib/Target/LoongArch/LoongArchLASXInstrInfo.td +++ b/llvm/lib/Target/LoongArch/LoongArchLASXInstrInfo.td @@ -1840,6 +1840,10 @@ def : Pat<(vt (concat_vectors LSX128:$vd, LSX128:$vj)), (XVPERMI_Q (SUBREG_TO_REG (i64 0), LSX128:$vd, sub_128), (SUBREG_TO_REG (i64 0), LSX128:$vj, sub_128), 2)>; +// XVABSD_{B/H/W/D}[U] +defm : PatXrXr; +defm : PatXrXrU; + } // Predicates = [HasExtLASX] /// Intrinsic pattern diff --git a/llvm/lib/Target/LoongArch/LoongArchLSXInstrInfo.td b/llvm/lib/Target/LoongArch/LoongArchLSXInstrInfo.td index b0d880749bf92..e7327ce7461f7 100644 --- a/llvm/lib/Target/LoongArch/LoongArchLSXInstrInfo.td +++ b/llvm/lib/Target/LoongArch/LoongArchLSXInstrInfo.td @@ -2002,6 +2002,10 @@ def : Pat<(f32 f32imm_vldi:$in), def : Pat<(f64 f64imm_vldi:$in), (f64 (EXTRACT_SUBREG (VLDI (to_f64imm_vldi f64imm_vldi:$in)), sub_64))>; +// VABSD_{B/H/W/D}[U] +defm : PatVrVr; +defm : PatVrVrU; + } // Predicates = [HasExtLSX] /// Intrinsic pattern diff --git a/llvm/test/CodeGen/LoongArch/lasx/ir-instruction/absd.ll b/llvm/test/CodeGen/LoongArch/lasx/ir-instruction/absd.ll index bd5b16f5147a2..c5df9f8420837 100644 --- a/llvm/test/CodeGen/LoongArch/lasx/ir-instruction/absd.ll +++ b/llvm/test/CodeGen/LoongArch/lasx/ir-instruction/absd.ll @@ -1,7 +1,6 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py ; RUN: llc -mtriple=loongarch64 -mattr=+lasx < %s | FileCheck %s -;; TODO: Currently LoongArch generates sub-optimal code for these cases ;; 1. trunc(abs(sub(sext(a),sext(b)))) -> abds(a,b) or abdu(a,b) ;; 2. abs(sub_nsw(x, y)) -> abds(a,b) ;; 3. sub(smax(a,b),smin(a,b)) -> abds(a,b) or abdu(a,b) @@ -9,16 +8,12 @@ ;; 5. sub(select(icmp(a,b),a,b),select(icmp(a,b),b,a)) -> abds(a,b) or abdu(a,b) ;; ;; abds / abdu can be lowered to xvabsd.{b/h/w/d} / xvabsd.{b/h/w/d}u instruction. -;; -;; Later patch will address it. ;; trunc(abs(sub(sext(a),sext(b)))) -> abds(a,b) define <32 x i8> @xvabsd_b(<32 x i8> %a, <32 x i8> %b) { ; CHECK-LABEL: xvabsd_b: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.b $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.b $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.b $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.sext = sext <32 x i8> %a to <32 x i16> %b.sext = sext <32 x i8> %b to <32 x i16> @@ -31,9 +26,7 @@ define <32 x i8> @xvabsd_b(<32 x i8> %a, <32 x i8> %b) { define <16 x i16> @xvabsd_h(<16 x i16> %a, <16 x i16> %b) { ; CHECK-LABEL: xvabsd_h: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.h $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.h $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.h $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.sext = sext <16 x i16> %a to <16 x i32> %b.sext = sext <16 x i16> %b to <16 x i32> @@ -46,9 +39,7 @@ define <16 x i16> @xvabsd_h(<16 x i16> %a, <16 x i16> %b) { define <8 x i32> @xvabsd_w(<8 x i32> %a, <8 x i32> %b) { ; CHECK-LABEL: xvabsd_w: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.w $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.w $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.w $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.sext = sext <8 x i32> %a to <8 x i64> %b.sext = sext <8 x i32> %b to <8 x i64> @@ -61,9 +52,7 @@ define <8 x i32> @xvabsd_w(<8 x i32> %a, <8 x i32> %b) { define <4 x i64> @xvabsd_d(<4 x i64> %a, <4 x i64> %b) { ; CHECK-LABEL: xvabsd_d: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.d $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.d $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.d $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.sext = sext <4 x i64> %a to <4 x i128> %b.sext = sext <4 x i64> %b to <4 x i128> @@ -76,9 +65,7 @@ define <4 x i64> @xvabsd_d(<4 x i64> %a, <4 x i64> %b) { define <32 x i8> @xvabsd_bu(<32 x i8> %a, <32 x i8> %b) { ; CHECK-LABEL: xvabsd_bu: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.bu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.bu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.bu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.zext = zext <32 x i8> %a to <32 x i16> %b.zext = zext <32 x i8> %b to <32 x i16> @@ -91,9 +78,7 @@ define <32 x i8> @xvabsd_bu(<32 x i8> %a, <32 x i8> %b) { define <16 x i16> @xvabsd_hu(<16 x i16> %a, <16 x i16> %b) { ; CHECK-LABEL: xvabsd_hu: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.hu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.hu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.hu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.zext = zext <16 x i16> %a to <16 x i32> %b.zext = zext <16 x i16> %b to <16 x i32> @@ -106,9 +91,7 @@ define <16 x i16> @xvabsd_hu(<16 x i16> %a, <16 x i16> %b) { define <8 x i32> @xvabsd_wu(<8 x i32> %a, <8 x i32> %b) { ; CHECK-LABEL: xvabsd_wu: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.wu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.wu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.wu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.zext = zext <8 x i32> %a to <8 x i64> %b.zext = zext <8 x i32> %b to <8 x i64> @@ -121,9 +104,7 @@ define <8 x i32> @xvabsd_wu(<8 x i32> %a, <8 x i32> %b) { define <4 x i64> @xvabsd_du(<4 x i64> %a, <4 x i64> %b) { ; CHECK-LABEL: xvabsd_du: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.du $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.du $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.du $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a.zext = zext <4 x i64> %a to <4 x i128> %b.zext = zext <4 x i64> %b to <4 x i128> @@ -137,9 +118,7 @@ define <4 x i64> @xvabsd_du(<4 x i64> %a, <4 x i64> %b) { define <32 x i8> @xvabsd_b_nsw(<32 x i8> %a, <32 x i8> %b) { ; CHECK-LABEL: xvabsd_b_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr1 -; CHECK-NEXT: xvneg.b $xr1, $xr0 -; CHECK-NEXT: xvmax.b $xr0, $xr0, $xr1 +; CHECK-NEXT: xvabsd.b $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %sub = sub nsw <32 x i8> %a, %b %abs = call <32 x i8> @llvm.abs.v32i8(<32 x i8> %sub, i1 true) @@ -149,9 +128,7 @@ define <32 x i8> @xvabsd_b_nsw(<32 x i8> %a, <32 x i8> %b) { define <16 x i16> @xvabsd_h_nsw(<16 x i16> %a, <16 x i16> %b) { ; CHECK-LABEL: xvabsd_h_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr1 -; CHECK-NEXT: xvneg.h $xr1, $xr0 -; CHECK-NEXT: xvmax.h $xr0, $xr0, $xr1 +; CHECK-NEXT: xvabsd.h $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %sub = sub nsw <16 x i16> %a, %b %abs = call <16 x i16> @llvm.abs.v16i16(<16 x i16> %sub, i1 true) @@ -161,9 +138,7 @@ define <16 x i16> @xvabsd_h_nsw(<16 x i16> %a, <16 x i16> %b) { define <8 x i32> @xvabsd_w_nsw(<8 x i32> %a, <8 x i32> %b) { ; CHECK-LABEL: xvabsd_w_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr1 -; CHECK-NEXT: xvneg.w $xr1, $xr0 -; CHECK-NEXT: xvmax.w $xr0, $xr0, $xr1 +; CHECK-NEXT: xvabsd.w $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %sub = sub nsw <8 x i32> %a, %b %abs = call <8 x i32> @llvm.abs.v8i32(<8 x i32> %sub, i1 true) @@ -173,9 +148,7 @@ define <8 x i32> @xvabsd_w_nsw(<8 x i32> %a, <8 x i32> %b) { define <4 x i64> @xvabsd_d_nsw(<4 x i64> %a, <4 x i64> %b) { ; CHECK-LABEL: xvabsd_d_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr1 -; CHECK-NEXT: xvneg.d $xr1, $xr0 -; CHECK-NEXT: xvmax.d $xr0, $xr0, $xr1 +; CHECK-NEXT: xvabsd.d $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %sub = sub nsw <4 x i64> %a, %b %abs = call <4 x i64> @llvm.abs.v4i64(<4 x i64> %sub, i1 true) @@ -186,9 +159,7 @@ define <4 x i64> @xvabsd_d_nsw(<4 x i64> %a, <4 x i64> %b) { define <32 x i8> @maxmin_b(<32 x i8> %0, <32 x i8> %1) { ; CHECK-LABEL: maxmin_b: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.b $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.b $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.b $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <32 x i8> @llvm.smax.v32i8(<32 x i8> %0, <32 x i8> %1) %b = tail call <32 x i8> @llvm.smin.v32i8(<32 x i8> %0, <32 x i8> %1) @@ -199,9 +170,7 @@ define <32 x i8> @maxmin_b(<32 x i8> %0, <32 x i8> %1) { define <16 x i16> @maxmin_h(<16 x i16> %0, <16 x i16> %1) { ; CHECK-LABEL: maxmin_h: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.h $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.h $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.h $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <16 x i16> @llvm.smax.v16i16(<16 x i16> %0, <16 x i16> %1) %b = tail call <16 x i16> @llvm.smin.v16i16(<16 x i16> %0, <16 x i16> %1) @@ -212,9 +181,7 @@ define <16 x i16> @maxmin_h(<16 x i16> %0, <16 x i16> %1) { define <8 x i32> @maxmin_w(<8 x i32> %0, <8 x i32> %1) { ; CHECK-LABEL: maxmin_w: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.w $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.w $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.w $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <8 x i32> @llvm.smax.v8i32(<8 x i32> %0, <8 x i32> %1) %b = tail call <8 x i32> @llvm.smin.v8i32(<8 x i32> %0, <8 x i32> %1) @@ -225,9 +192,7 @@ define <8 x i32> @maxmin_w(<8 x i32> %0, <8 x i32> %1) { define <4 x i64> @maxmin_d(<4 x i64> %0, <4 x i64> %1) { ; CHECK-LABEL: maxmin_d: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.d $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.d $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.d $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <4 x i64> @llvm.smax.v4i64(<4 x i64> %0, <4 x i64> %1) %b = tail call <4 x i64> @llvm.smin.v4i64(<4 x i64> %0, <4 x i64> %1) @@ -238,9 +203,7 @@ define <4 x i64> @maxmin_d(<4 x i64> %0, <4 x i64> %1) { define <32 x i8> @maxmin_bu(<32 x i8> %0, <32 x i8> %1) { ; CHECK-LABEL: maxmin_bu: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.bu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.bu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.bu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <32 x i8> @llvm.umax.v32i8(<32 x i8> %0, <32 x i8> %1) %b = tail call <32 x i8> @llvm.umin.v32i8(<32 x i8> %0, <32 x i8> %1) @@ -251,9 +214,7 @@ define <32 x i8> @maxmin_bu(<32 x i8> %0, <32 x i8> %1) { define <16 x i16> @maxmin_hu(<16 x i16> %0, <16 x i16> %1) { ; CHECK-LABEL: maxmin_hu: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.hu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.hu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.hu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <16 x i16> @llvm.umax.v16i16(<16 x i16> %0, <16 x i16> %1) %b = tail call <16 x i16> @llvm.umin.v16i16(<16 x i16> %0, <16 x i16> %1) @@ -264,9 +225,7 @@ define <16 x i16> @maxmin_hu(<16 x i16> %0, <16 x i16> %1) { define <8 x i32> @maxmin_wu(<8 x i32> %0, <8 x i32> %1) { ; CHECK-LABEL: maxmin_wu: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.wu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.wu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.wu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <8 x i32> @llvm.umax.v8i32(<8 x i32> %0, <8 x i32> %1) %b = tail call <8 x i32> @llvm.umin.v8i32(<8 x i32> %0, <8 x i32> %1) @@ -277,9 +236,7 @@ define <8 x i32> @maxmin_wu(<8 x i32> %0, <8 x i32> %1) { define <4 x i64> @maxmin_du(<4 x i64> %0, <4 x i64> %1) { ; CHECK-LABEL: maxmin_du: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.du $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.du $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.du $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <4 x i64> @llvm.umax.v4i64(<4 x i64> %0, <4 x i64> %1) %b = tail call <4 x i64> @llvm.umin.v4i64(<4 x i64> %0, <4 x i64> %1) @@ -290,9 +247,7 @@ define <4 x i64> @maxmin_du(<4 x i64> %0, <4 x i64> %1) { define <32 x i8> @maxmin_bu_com1(<32 x i8> %0, <32 x i8> %1) { ; CHECK-LABEL: maxmin_bu_com1: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.bu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.bu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.bu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %a = tail call <32 x i8> @llvm.umax.v32i8(<32 x i8> %0, <32 x i8> %1) %b = tail call <32 x i8> @llvm.umin.v32i8(<32 x i8> %1, <32 x i8> %0) @@ -304,9 +259,7 @@ define <32 x i8> @maxmin_bu_com1(<32 x i8> %0, <32 x i8> %1) { define <32 x i8> @xvabsd_b_cmp(<32 x i8> %a, <32 x i8> %b) nounwind { ; CHECK-LABEL: xvabsd_b_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.b $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.b $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.b $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp slt <32 x i8> %a, %b %ab = sub <32 x i8> %a, %b @@ -318,9 +271,7 @@ define <32 x i8> @xvabsd_b_cmp(<32 x i8> %a, <32 x i8> %b) nounwind { define <16 x i16> @xvabsd_h_cmp(<16 x i16> %a, <16 x i16> %b) nounwind { ; CHECK-LABEL: xvabsd_h_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.h $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.h $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.h $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp slt <16 x i16> %a, %b %ab = sub <16 x i16> %a, %b @@ -332,9 +283,7 @@ define <16 x i16> @xvabsd_h_cmp(<16 x i16> %a, <16 x i16> %b) nounwind { define <8 x i32> @xvabsd_w_cmp(<8 x i32> %a, <8 x i32> %b) nounwind { ; CHECK-LABEL: xvabsd_w_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.w $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.w $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.w $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp slt <8 x i32> %a, %b %ab = sub <8 x i32> %a, %b @@ -346,9 +295,7 @@ define <8 x i32> @xvabsd_w_cmp(<8 x i32> %a, <8 x i32> %b) nounwind { define <4 x i64> @xvabsd_d_cmp(<4 x i64> %a, <4 x i64> %b) nounwind { ; CHECK-LABEL: xvabsd_d_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.d $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.d $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.d $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp slt <4 x i64> %a, %b %ab = sub <4 x i64> %a, %b @@ -360,9 +307,7 @@ define <4 x i64> @xvabsd_d_cmp(<4 x i64> %a, <4 x i64> %b) nounwind { define <32 x i8> @xvabsd_bu_cmp(<32 x i8> %a, <32 x i8> %b) nounwind { ; CHECK-LABEL: xvabsd_bu_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.bu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.bu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.bu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp ult <32 x i8> %a, %b %ab = sub <32 x i8> %a, %b @@ -374,9 +319,7 @@ define <32 x i8> @xvabsd_bu_cmp(<32 x i8> %a, <32 x i8> %b) nounwind { define <16 x i16> @xvabsd_hu_cmp(<16 x i16> %a, <16 x i16> %b) nounwind { ; CHECK-LABEL: xvabsd_hu_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.hu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.hu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.hu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp ult <16 x i16> %a, %b %ab = sub <16 x i16> %a, %b @@ -388,9 +331,7 @@ define <16 x i16> @xvabsd_hu_cmp(<16 x i16> %a, <16 x i16> %b) nounwind { define <8 x i32> @xvabsd_wu_cmp(<8 x i32> %a, <8 x i32> %b) nounwind { ; CHECK-LABEL: xvabsd_wu_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.wu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.wu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.wu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp ult <8 x i32> %a, %b %ab = sub <8 x i32> %a, %b @@ -402,9 +343,7 @@ define <8 x i32> @xvabsd_wu_cmp(<8 x i32> %a, <8 x i32> %b) nounwind { define <4 x i64> @xvabsd_du_cmp(<4 x i64> %a, <4 x i64> %b) nounwind { ; CHECK-LABEL: xvabsd_du_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.du $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.du $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.du $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp ult <4 x i64> %a, %b %ab = sub <4 x i64> %a, %b @@ -417,9 +356,7 @@ define <4 x i64> @xvabsd_du_cmp(<4 x i64> %a, <4 x i64> %b) nounwind { define <32 x i8> @xvabsd_b_select(<32 x i8> %a, <32 x i8> %b) nounwind { ; CHECK-LABEL: xvabsd_b_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.b $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.b $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.b $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp slt <32 x i8> %a, %b %ab = select <32 x i1> %cmp, <32 x i8> %a, <32 x i8> %b @@ -431,9 +368,7 @@ define <32 x i8> @xvabsd_b_select(<32 x i8> %a, <32 x i8> %b) nounwind { define <16 x i16> @xvabsd_h_select(<16 x i16> %a, <16 x i16> %b) nounwind { ; CHECK-LABEL: xvabsd_h_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.h $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.h $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.h $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp sle <16 x i16> %a, %b %ab = select <16 x i1> %cmp, <16 x i16> %a, <16 x i16> %b @@ -445,9 +380,7 @@ define <16 x i16> @xvabsd_h_select(<16 x i16> %a, <16 x i16> %b) nounwind { define <8 x i32> @xvabsd_w_select(<8 x i32> %a, <8 x i32> %b) nounwind { ; CHECK-LABEL: xvabsd_w_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.w $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.w $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.w $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp sgt <8 x i32> %a, %b %ab = select <8 x i1> %cmp, <8 x i32> %a, <8 x i32> %b @@ -459,9 +392,7 @@ define <8 x i32> @xvabsd_w_select(<8 x i32> %a, <8 x i32> %b) nounwind { define <4 x i64> @xvabsd_d_select(<4 x i64> %a, <4 x i64> %b) nounwind { ; CHECK-LABEL: xvabsd_d_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.d $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.d $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.d $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp sge <4 x i64> %a, %b %ab = select <4 x i1> %cmp, <4 x i64> %a, <4 x i64> %b @@ -473,9 +404,7 @@ define <4 x i64> @xvabsd_d_select(<4 x i64> %a, <4 x i64> %b) nounwind { define <32 x i8> @xvabsd_bu_select(<32 x i8> %a, <32 x i8> %b) nounwind { ; CHECK-LABEL: xvabsd_bu_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.bu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.bu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.b $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.bu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp ult <32 x i8> %a, %b %ab = select <32 x i1> %cmp, <32 x i8> %a, <32 x i8> %b @@ -487,9 +416,7 @@ define <32 x i8> @xvabsd_bu_select(<32 x i8> %a, <32 x i8> %b) nounwind { define <16 x i16> @xvabsd_hu_select(<16 x i16> %a, <16 x i16> %b) nounwind { ; CHECK-LABEL: xvabsd_hu_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.hu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.hu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.h $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.hu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp ule <16 x i16> %a, %b %ab = select <16 x i1> %cmp, <16 x i16> %a, <16 x i16> %b @@ -501,9 +428,7 @@ define <16 x i16> @xvabsd_hu_select(<16 x i16> %a, <16 x i16> %b) nounwind { define <8 x i32> @xvabsd_wu_select(<8 x i32> %a, <8 x i32> %b) nounwind { ; CHECK-LABEL: xvabsd_wu_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.wu $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.wu $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.w $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.wu $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp ugt <8 x i32> %a, %b %ab = select <8 x i1> %cmp, <8 x i32> %a, <8 x i32> %b @@ -515,9 +440,7 @@ define <8 x i32> @xvabsd_wu_select(<8 x i32> %a, <8 x i32> %b) nounwind { define <4 x i64> @xvabsd_du_select(<4 x i64> %a, <4 x i64> %b) nounwind { ; CHECK-LABEL: xvabsd_du_select: ; CHECK: # %bb.0: -; CHECK-NEXT: xvmin.du $xr2, $xr0, $xr1 -; CHECK-NEXT: xvmax.du $xr0, $xr0, $xr1 -; CHECK-NEXT: xvsub.d $xr0, $xr0, $xr2 +; CHECK-NEXT: xvabsd.du $xr0, $xr0, $xr1 ; CHECK-NEXT: ret %cmp = icmp uge <4 x i64> %a, %b %ab = select <4 x i1> %cmp, <4 x i64> %a, <4 x i64> %b diff --git a/llvm/test/CodeGen/LoongArch/lsx/ir-instruction/absd.ll b/llvm/test/CodeGen/LoongArch/lsx/ir-instruction/absd.ll index 2cbd74204d5d6..f77a31b600761 100644 --- a/llvm/test/CodeGen/LoongArch/lsx/ir-instruction/absd.ll +++ b/llvm/test/CodeGen/LoongArch/lsx/ir-instruction/absd.ll @@ -1,7 +1,6 @@ ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py ; RUN: llc -mtriple=loongarch64 -mattr=+lsx < %s | FileCheck %s -;; TODO: Currently LoongArch generates sub-optimal code for five cases ;; 1. trunc(abs(sub(sext(a),sext(b)))) -> abds(a,b) or abdu(a,b) ;; 2. abs(sub_nsw(x, y)) -> abds(a,b) ;; 3. sub(smax(a,b),smin(a,b)) -> abds(a,b) or abdu(a,b) @@ -9,16 +8,12 @@ ;; 5. sub(select(icmp(a,b),a,b),select(icmp(a,b),b,a)) -> abds(a,b) or abdu(a,b) ;; ;; abds / abdu can be lowered to vabsd.{b/h/w/d} / vabsd.{b/h/w/d}u instruction -;; -;; Later patch will address it. ;; trunc(abs(sub(sext(a),sext(b)))) -> abds(a,b) define <16 x i8> @vabsd_b(<16 x i8> %a, <16 x i8> %b) { ; CHECK-LABEL: vabsd_b: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.b $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.b $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.b $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.sext = sext <16 x i8> %a to <16 x i16> %b.sext = sext <16 x i8> %b to <16 x i16> @@ -31,9 +26,7 @@ define <16 x i8> @vabsd_b(<16 x i8> %a, <16 x i8> %b) { define <8 x i16> @vabsd_h(<8 x i16> %a, <8 x i16> %b) { ; CHECK-LABEL: vabsd_h: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.h $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.h $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.h $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.sext = sext <8 x i16> %a to <8 x i32> %b.sext = sext <8 x i16> %b to <8 x i32> @@ -46,9 +39,7 @@ define <8 x i16> @vabsd_h(<8 x i16> %a, <8 x i16> %b) { define <4 x i32> @vabsd_w(<4 x i32> %a, <4 x i32> %b) { ; CHECK-LABEL: vabsd_w: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.w $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.w $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.w $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.sext = sext <4 x i32> %a to <4 x i64> %b.sext = sext <4 x i32> %b to <4 x i64> @@ -61,9 +52,7 @@ define <4 x i32> @vabsd_w(<4 x i32> %a, <4 x i32> %b) { define <2 x i64> @vabsd_d(<2 x i64> %a, <2 x i64> %b) { ; CHECK-LABEL: vabsd_d: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.d $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.d $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.d $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.sext = sext <2 x i64> %a to <2 x i128> %b.sext = sext <2 x i64> %b to <2 x i128> @@ -76,9 +65,7 @@ define <2 x i64> @vabsd_d(<2 x i64> %a, <2 x i64> %b) { define <16 x i8> @vabsd_bu(<16 x i8> %a, <16 x i8> %b) { ; CHECK-LABEL: vabsd_bu: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.bu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.bu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.bu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.zext = zext <16 x i8> %a to <16 x i16> %b.zext = zext <16 x i8> %b to <16 x i16> @@ -91,9 +78,7 @@ define <16 x i8> @vabsd_bu(<16 x i8> %a, <16 x i8> %b) { define <8 x i16> @vabsd_hu(<8 x i16> %a, <8 x i16> %b) { ; CHECK-LABEL: vabsd_hu: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.hu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.hu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.hu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.zext = zext <8 x i16> %a to <8 x i32> %b.zext = zext <8 x i16> %b to <8 x i32> @@ -106,9 +91,7 @@ define <8 x i16> @vabsd_hu(<8 x i16> %a, <8 x i16> %b) { define <4 x i32> @vabsd_wu(<4 x i32> %a, <4 x i32> %b) { ; CHECK-LABEL: vabsd_wu: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.wu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.wu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.wu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.zext = zext <4 x i32> %a to <4 x i64> %b.zext = zext <4 x i32> %b to <4 x i64> @@ -121,9 +104,7 @@ define <4 x i32> @vabsd_wu(<4 x i32> %a, <4 x i32> %b) { define <2 x i64> @vabsd_du(<2 x i64> %a, <2 x i64> %b) { ; CHECK-LABEL: vabsd_du: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.du $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.du $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.du $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a.zext = zext <2 x i64> %a to <2 x i128> %b.zext = zext <2 x i64> %b to <2 x i128> @@ -137,9 +118,7 @@ define <2 x i64> @vabsd_du(<2 x i64> %a, <2 x i64> %b) { define <16 x i8> @vabsd_b_nsw(<16 x i8> %a, <16 x i8> %b) { ; CHECK-LABEL: vabsd_b_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr1 -; CHECK-NEXT: vneg.b $vr1, $vr0 -; CHECK-NEXT: vmax.b $vr0, $vr0, $vr1 +; CHECK-NEXT: vabsd.b $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %sub = sub nsw <16 x i8> %a, %b %abs = call <16 x i8> @llvm.abs.v16i8(<16 x i8> %sub, i1 true) @@ -149,9 +128,7 @@ define <16 x i8> @vabsd_b_nsw(<16 x i8> %a, <16 x i8> %b) { define <8 x i16> @vabsd_h_nsw(<8 x i16> %a, <8 x i16> %b) { ; CHECK-LABEL: vabsd_h_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr1 -; CHECK-NEXT: vneg.h $vr1, $vr0 -; CHECK-NEXT: vmax.h $vr0, $vr0, $vr1 +; CHECK-NEXT: vabsd.h $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %sub = sub nsw <8 x i16> %a, %b %abs = call <8 x i16> @llvm.abs.v8i16(<8 x i16> %sub, i1 true) @@ -161,9 +138,7 @@ define <8 x i16> @vabsd_h_nsw(<8 x i16> %a, <8 x i16> %b) { define <4 x i32> @vabsd_w_nsw(<4 x i32> %a, <4 x i32> %b) { ; CHECK-LABEL: vabsd_w_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr1 -; CHECK-NEXT: vneg.w $vr1, $vr0 -; CHECK-NEXT: vmax.w $vr0, $vr0, $vr1 +; CHECK-NEXT: vabsd.w $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %sub = sub nsw <4 x i32> %a, %b %abs = call <4 x i32> @llvm.abs.v4i32(<4 x i32> %sub, i1 true) @@ -173,9 +148,7 @@ define <4 x i32> @vabsd_w_nsw(<4 x i32> %a, <4 x i32> %b) { define <2 x i64> @vabsd_d_nsw(<2 x i64> %a, <2 x i64> %b) { ; CHECK-LABEL: vabsd_d_nsw: ; CHECK: # %bb.0: -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr1 -; CHECK-NEXT: vneg.d $vr1, $vr0 -; CHECK-NEXT: vmax.d $vr0, $vr0, $vr1 +; CHECK-NEXT: vabsd.d $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %sub = sub nsw <2 x i64> %a, %b %abs = call <2 x i64> @llvm.abs.v2i64(<2 x i64> %sub, i1 true) @@ -186,9 +159,7 @@ define <2 x i64> @vabsd_d_nsw(<2 x i64> %a, <2 x i64> %b) { define <16 x i8> @maxmin_b(<16 x i8> %0, <16 x i8> %1) { ; CHECK-LABEL: maxmin_b: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.b $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.b $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.b $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <16 x i8> @llvm.smax.v16i8(<16 x i8> %0, <16 x i8> %1) %b = tail call <16 x i8> @llvm.smin.v16i8(<16 x i8> %0, <16 x i8> %1) @@ -199,9 +170,7 @@ define <16 x i8> @maxmin_b(<16 x i8> %0, <16 x i8> %1) { define <8 x i16> @maxmin_h(<8 x i16> %0, <8 x i16> %1) { ; CHECK-LABEL: maxmin_h: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.h $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.h $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.h $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <8 x i16> @llvm.smax.v8i16(<8 x i16> %0, <8 x i16> %1) %b = tail call <8 x i16> @llvm.smin.v8i16(<8 x i16> %0, <8 x i16> %1) @@ -212,9 +181,7 @@ define <8 x i16> @maxmin_h(<8 x i16> %0, <8 x i16> %1) { define <4 x i32> @maxmin_w(<4 x i32> %0, <4 x i32> %1) { ; CHECK-LABEL: maxmin_w: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.w $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.w $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.w $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <4 x i32> @llvm.smax.v4i32(<4 x i32> %0, <4 x i32> %1) %b = tail call <4 x i32> @llvm.smin.v4i32(<4 x i32> %0, <4 x i32> %1) @@ -225,9 +192,7 @@ define <4 x i32> @maxmin_w(<4 x i32> %0, <4 x i32> %1) { define <2 x i64> @maxmin_d(<2 x i64> %0, <2 x i64> %1) { ; CHECK-LABEL: maxmin_d: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.d $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.d $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.d $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <2 x i64> @llvm.smax.v2i64(<2 x i64> %0, <2 x i64> %1) %b = tail call <2 x i64> @llvm.smin.v2i64(<2 x i64> %0, <2 x i64> %1) @@ -238,9 +203,7 @@ define <2 x i64> @maxmin_d(<2 x i64> %0, <2 x i64> %1) { define <16 x i8> @maxmin_bu(<16 x i8> %0, <16 x i8> %1) { ; CHECK-LABEL: maxmin_bu: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.bu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.bu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.bu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <16 x i8> @llvm.umax.v16i8(<16 x i8> %0, <16 x i8> %1) %b = tail call <16 x i8> @llvm.umin.v16i8(<16 x i8> %0, <16 x i8> %1) @@ -251,9 +214,7 @@ define <16 x i8> @maxmin_bu(<16 x i8> %0, <16 x i8> %1) { define <8 x i16> @maxmin_hu(<8 x i16> %0, <8 x i16> %1) { ; CHECK-LABEL: maxmin_hu: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.hu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.hu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.hu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <8 x i16> @llvm.umax.v8i16(<8 x i16> %0, <8 x i16> %1) %b = tail call <8 x i16> @llvm.umin.v8i16(<8 x i16> %0, <8 x i16> %1) @@ -264,9 +225,7 @@ define <8 x i16> @maxmin_hu(<8 x i16> %0, <8 x i16> %1) { define <4 x i32> @maxmin_wu(<4 x i32> %0, <4 x i32> %1) { ; CHECK-LABEL: maxmin_wu: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.wu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.wu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.wu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <4 x i32> @llvm.umax.v4i32(<4 x i32> %0, <4 x i32> %1) %b = tail call <4 x i32> @llvm.umin.v4i32(<4 x i32> %0, <4 x i32> %1) @@ -277,9 +236,7 @@ define <4 x i32> @maxmin_wu(<4 x i32> %0, <4 x i32> %1) { define <2 x i64> @maxmin_du(<2 x i64> %0, <2 x i64> %1) { ; CHECK-LABEL: maxmin_du: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.du $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.du $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.du $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <2 x i64> @llvm.umax.v2i64(<2 x i64> %0, <2 x i64> %1) %b = tail call <2 x i64> @llvm.umin.v2i64(<2 x i64> %0, <2 x i64> %1) @@ -290,9 +247,7 @@ define <2 x i64> @maxmin_du(<2 x i64> %0, <2 x i64> %1) { define <16 x i8> @maxmin_bu_com1(<16 x i8> %0, <16 x i8> %1) { ; CHECK-LABEL: maxmin_bu_com1: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.bu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.bu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.bu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %a = tail call <16 x i8> @llvm.umax.v16i8(<16 x i8> %0, <16 x i8> %1) %b = tail call <16 x i8> @llvm.umin.v16i8(<16 x i8> %1, <16 x i8> %0) @@ -304,9 +259,7 @@ define <16 x i8> @maxmin_bu_com1(<16 x i8> %0, <16 x i8> %1) { define <16 x i8> @vabsd_b_cmp(<16 x i8> %a, <16 x i8> %b) nounwind { ; CHECK-LABEL: vabsd_b_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.b $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.b $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.b $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp slt <16 x i8> %a, %b %ab = sub <16 x i8> %a, %b @@ -318,9 +271,7 @@ define <16 x i8> @vabsd_b_cmp(<16 x i8> %a, <16 x i8> %b) nounwind { define <8 x i16> @vabsd_h_cmp(<8 x i16> %a, <8 x i16> %b) nounwind { ; CHECK-LABEL: vabsd_h_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.h $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.h $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.h $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp slt <8 x i16> %a, %b %ab = sub <8 x i16> %a, %b @@ -332,9 +283,7 @@ define <8 x i16> @vabsd_h_cmp(<8 x i16> %a, <8 x i16> %b) nounwind { define <4 x i32> @vabsd_w_cmp(<4 x i32> %a, <4 x i32> %b) nounwind { ; CHECK-LABEL: vabsd_w_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.w $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.w $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.w $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp slt <4 x i32> %a, %b %ab = sub <4 x i32> %a, %b @@ -346,9 +295,7 @@ define <4 x i32> @vabsd_w_cmp(<4 x i32> %a, <4 x i32> %b) nounwind { define <2 x i64> @vabsd_d_cmp(<2 x i64> %a, <2 x i64> %b) nounwind { ; CHECK-LABEL: vabsd_d_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.d $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.d $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.d $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp slt <2 x i64> %a, %b %ab = sub <2 x i64> %a, %b @@ -360,9 +307,7 @@ define <2 x i64> @vabsd_d_cmp(<2 x i64> %a, <2 x i64> %b) nounwind { define <16 x i8> @vabsd_bu_cmp(<16 x i8> %a, <16 x i8> %b) nounwind { ; CHECK-LABEL: vabsd_bu_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.bu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.bu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.bu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp ult <16 x i8> %a, %b %ab = sub <16 x i8> %a, %b @@ -374,9 +319,7 @@ define <16 x i8> @vabsd_bu_cmp(<16 x i8> %a, <16 x i8> %b) nounwind { define <8 x i16> @vabsd_hu_cmp(<8 x i16> %a, <8 x i16> %b) nounwind { ; CHECK-LABEL: vabsd_hu_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.hu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.hu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.hu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp ult <8 x i16> %a, %b %ab = sub <8 x i16> %a, %b @@ -388,9 +331,7 @@ define <8 x i16> @vabsd_hu_cmp(<8 x i16> %a, <8 x i16> %b) nounwind { define <4 x i32> @vabsd_wu_cmp(<4 x i32> %a, <4 x i32> %b) nounwind { ; CHECK-LABEL: vabsd_wu_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.wu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.wu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.wu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp ult <4 x i32> %a, %b %ab = sub <4 x i32> %a, %b @@ -402,9 +343,7 @@ define <4 x i32> @vabsd_wu_cmp(<4 x i32> %a, <4 x i32> %b) nounwind { define <2 x i64> @vabsd_du_cmp(<2 x i64> %a, <2 x i64> %b) nounwind { ; CHECK-LABEL: vabsd_du_cmp: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.du $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.du $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.du $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp ult <2 x i64> %a, %b %ab = sub <2 x i64> %a, %b @@ -417,9 +356,7 @@ define <2 x i64> @vabsd_du_cmp(<2 x i64> %a, <2 x i64> %b) nounwind { define <16 x i8> @vabsd_b_select(<16 x i8> %a, <16 x i8> %b) nounwind { ; CHECK-LABEL: vabsd_b_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.b $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.b $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.b $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp slt <16 x i8> %a, %b %ab = select <16 x i1> %cmp, <16 x i8> %a, <16 x i8> %b @@ -431,9 +368,7 @@ define <16 x i8> @vabsd_b_select(<16 x i8> %a, <16 x i8> %b) nounwind { define <8 x i16> @vabsd_h_select(<8 x i16> %a, <8 x i16> %b) nounwind { ; CHECK-LABEL: vabsd_h_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.h $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.h $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.h $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp sle <8 x i16> %a, %b %ab = select <8 x i1> %cmp, <8 x i16> %a, <8 x i16> %b @@ -445,9 +380,7 @@ define <8 x i16> @vabsd_h_select(<8 x i16> %a, <8 x i16> %b) nounwind { define <4 x i32> @vabsd_w_select(<4 x i32> %a, <4 x i32> %b) nounwind { ; CHECK-LABEL: vabsd_w_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.w $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.w $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.w $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp sgt <4 x i32> %a, %b %ab = select <4 x i1> %cmp, <4 x i32> %a, <4 x i32> %b @@ -459,9 +392,7 @@ define <4 x i32> @vabsd_w_select(<4 x i32> %a, <4 x i32> %b) nounwind { define <2 x i64> @vabsd_d_select(<2 x i64> %a, <2 x i64> %b) nounwind { ; CHECK-LABEL: vabsd_d_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.d $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.d $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.d $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp sge <2 x i64> %a, %b %ab = select <2 x i1> %cmp, <2 x i64> %a, <2 x i64> %b @@ -473,9 +404,7 @@ define <2 x i64> @vabsd_d_select(<2 x i64> %a, <2 x i64> %b) nounwind { define <16 x i8> @vabsd_bu_select(<16 x i8> %a, <16 x i8> %b) nounwind { ; CHECK-LABEL: vabsd_bu_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.bu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.bu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.b $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.bu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp ult <16 x i8> %a, %b %ab = select <16 x i1> %cmp, <16 x i8> %a, <16 x i8> %b @@ -487,9 +416,7 @@ define <16 x i8> @vabsd_bu_select(<16 x i8> %a, <16 x i8> %b) nounwind { define <8 x i16> @vabsd_hu_select(<8 x i16> %a, <8 x i16> %b) nounwind { ; CHECK-LABEL: vabsd_hu_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.hu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.hu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.h $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.hu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp ule <8 x i16> %a, %b %ab = select <8 x i1> %cmp, <8 x i16> %a, <8 x i16> %b @@ -501,9 +428,7 @@ define <8 x i16> @vabsd_hu_select(<8 x i16> %a, <8 x i16> %b) nounwind { define <4 x i32> @vabsd_wu_select(<4 x i32> %a, <4 x i32> %b) nounwind { ; CHECK-LABEL: vabsd_wu_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.wu $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.wu $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.w $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.wu $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp ugt <4 x i32> %a, %b %ab = select <4 x i1> %cmp, <4 x i32> %a, <4 x i32> %b @@ -515,9 +440,7 @@ define <4 x i32> @vabsd_wu_select(<4 x i32> %a, <4 x i32> %b) nounwind { define <2 x i64> @vabsd_du_select(<2 x i64> %a, <2 x i64> %b) nounwind { ; CHECK-LABEL: vabsd_du_select: ; CHECK: # %bb.0: -; CHECK-NEXT: vmin.du $vr2, $vr0, $vr1 -; CHECK-NEXT: vmax.du $vr0, $vr0, $vr1 -; CHECK-NEXT: vsub.d $vr0, $vr0, $vr2 +; CHECK-NEXT: vabsd.du $vr0, $vr0, $vr1 ; CHECK-NEXT: ret %cmp = icmp uge <2 x i64> %a, %b %ab = select <2 x i1> %cmp, <2 x i64> %a, <2 x i64> %b From b07c88563febdb62b82daad0480d7b6131bc54d4 Mon Sep 17 00:00:00 2001 From: Jakub Kuderski Date: Tue, 15 Apr 2025 23:44:33 -0400 Subject: [PATCH 61/63] [Support] Add format object for interleaved ranges (#135517) Add two new format functions for printing ranges: `interleaved` and `interleaved_array`. This is meant to improve the ergonomics of printing ranges. Before this patch, we have to either use `llvm::interleave` or write a for loop by hand. For example: Before: ```c++ ArrayRef types = ...; ArrayRef values = ...; LLVM_DEBUG({ llvm::dbgs() << "Types: "; llvm::interleave_comma(llvm::dbgs(), types); llvm::dbgs() << "\n"; llvm::dbgs() << "Values: ["; llvm::interleave_comma(llvm::dbgs(), values); llvm::dbgs() << "]\n"; }): ``` After: ```c++ ArrayRef types = ...; ArrayRef values = ...; LLVM_DEBUG(llvm::dbgs() << "Types: " << interleaved(types) << "\n"); LLVM_DEBUG(llvm::dbgs() << "Values: " << interleaved_array(values) << "\n"); ``` The separator and the prefix/suffix strings are customizable. --- llvm/include/llvm/Support/InterleavedRange.h | 99 +++++++++++++++++++ llvm/unittests/Support/CMakeLists.txt | 3 +- .../Support/InterleavedRangeTest.cpp | 70 +++++++++++++ 3 files changed, 171 insertions(+), 1 deletion(-) create mode 100644 llvm/include/llvm/Support/InterleavedRange.h create mode 100644 llvm/unittests/Support/InterleavedRangeTest.cpp diff --git a/llvm/include/llvm/Support/InterleavedRange.h b/llvm/include/llvm/Support/InterleavedRange.h new file mode 100644 index 0000000000000..4e70028504806 --- /dev/null +++ b/llvm/include/llvm/Support/InterleavedRange.h @@ -0,0 +1,99 @@ +//===- InterleavedRange.h - Output stream formatting for ranges -----------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// Implements format objects for printing ranges to output streams. +// For example: +// ```c++ +// ArrayRef Types = ...; +// OS << "Types: " << interleaved(Types); // ==> "Types: i32, f16, i8" +// ArrayRef Values = ...; +// OS << "Values: " << interleaved_array(Values); // ==> "Values: [1, 2, 3]" +// ``` +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_SUPPORT_INTERLEAVED_RANGE_H +#define LLVM_SUPPORT_INTERLEAVED_RANGE_H + +#include "llvm/ADT/STLExtras.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/raw_ostream.h" + +namespace llvm { + +/// Format object class for interleaved ranges. Supports specifying the +/// separator and, optionally, the prefix and suffix to be printed surrounding +/// the range. +/// Uses the operator '<<' of the range element type for printing. The range +/// type itself does not have to have an '<<' operator defined. +template class InterleavedRange { + const Range &TheRange; + StringRef Separator; + StringRef Prefix; + StringRef Suffix; + +public: + InterleavedRange(const Range &R, StringRef Separator, StringRef Prefix, + StringRef Suffix) + : TheRange(R), Separator(Separator), Prefix(Prefix), Suffix(Suffix) {} + + friend raw_ostream &operator<<(raw_ostream &OS, + const InterleavedRange &Interleaved) { + if (!Interleaved.Prefix.empty()) + OS << Interleaved.Prefix; + llvm::interleave(Interleaved.TheRange, OS, Interleaved.Separator); + if (!Interleaved.Suffix.empty()) + OS << Interleaved.Suffix; + return OS; + } + + std::string str() const { + std::string Result; + raw_string_ostream Stream(Result); + Stream << *this; + Stream.flush(); + return Result; + } + + operator std::string() const { return str(); } +}; + +/// Output range `R` as a sequence of interleaved elements. Requires the range +/// element type to be printable using `raw_ostream& operator<<`. The +/// `Separator` and `Prefix` / `Suffix` can be customized. Examples: +/// ```c++ +/// SmallVector Vals = {1, 2, 3}; +/// OS << interleaved(Vals); // ==> "1, 2, 3" +/// OS << interleaved(Vals, ";"); // ==> "1;2;3" +/// OS << interleaved(Vals, " ", "{", "}"); // ==> "{1 2 3}" +/// ``` +template +InterleavedRange interleaved(const Range &R, StringRef Separator = ", ", + StringRef Prefix = "", + StringRef Suffix = "") { + return {R, Separator, Prefix, Suffix}; +} + +/// Output range `R` as an array of interleaved elements. Requires the range +/// element type to be printable using `raw_ostream& operator<<`. The +/// `Separator` can be customized. Examples: +/// ```c++ +/// SmallVector Vals = {1, 2, 3}; +/// OS << interleaved_array(Vals); // ==> "[1, 2, 3]" +/// OS << interleaved_array(Vals, ";"); // ==> "[1;2;3]" +/// OS << interleaved_array(Vals, " "); // ==> "[1 2 3]" +/// ``` +template +InterleavedRange interleaved_array(const Range &R, + StringRef Separator = ", ") { + return {R, Separator, "[", "]"}; +} + +} // end namespace llvm + +#endif // LLVM_SUPPORT_INTERLEAVED_RANGE_H diff --git a/llvm/unittests/Support/CMakeLists.txt b/llvm/unittests/Support/CMakeLists.txt index 6c4e7cb689b20..4a12a928af119 100644 --- a/llvm/unittests/Support/CMakeLists.txt +++ b/llvm/unittests/Support/CMakeLists.txt @@ -49,6 +49,7 @@ add_llvm_unittest(SupportTests HashBuilderTest.cpp IndexedAccessorTest.cpp InstructionCostTest.cpp + InterleavedRangeTest.cpp JSONTest.cpp KnownBitsTest.cpp LEB128Test.cpp @@ -61,7 +62,7 @@ add_llvm_unittest(SupportTests MemoryBufferRefTest.cpp MemoryBufferTest.cpp MemoryTest.cpp - MustacheTest.cpp + MustacheTest.cpp ModRefTest.cpp NativeFormatTests.cpp OptimizedStructLayoutTest.cpp diff --git a/llvm/unittests/Support/InterleavedRangeTest.cpp b/llvm/unittests/Support/InterleavedRangeTest.cpp new file mode 100644 index 0000000000000..8640b81fe8ad8 --- /dev/null +++ b/llvm/unittests/Support/InterleavedRangeTest.cpp @@ -0,0 +1,70 @@ +//===- InterleavedRangeTest.cpp - Unit tests for interleaved format -------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "llvm/Support/InterleavedRange.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/raw_ostream.h" +#include "gmock/gmock.h" +#include "gtest/gtest.h" + +using namespace llvm; + +namespace { + +TEST(InterleavedRangeTest, VectorInt) { + SmallVector V = {0, 1, 2, 3}; + + // First, make sure that the raw print API works as expected. + std::string Buff; + raw_string_ostream OS(Buff); + OS << interleaved(V); + EXPECT_EQ("0, 1, 2, 3", Buff); + Buff.clear(); + OS << interleaved_array(V); + EXPECT_EQ("[0, 1, 2, 3]", Buff); + + // In the rest of the tests, use `.str()` for convenience. + EXPECT_EQ("0, 1, 2, 3", interleaved(V).str()); + EXPECT_EQ("{{0,1,2,3}}", interleaved(V, ",", "{{", "}}").str()); + EXPECT_EQ("[0, 1, 2, 3]", interleaved_array(V).str()); + EXPECT_EQ("[0;1;2;3]", interleaved_array(V, ";").str()); + EXPECT_EQ("0;1;2;3", interleaved(V, ";").str()); +} + +TEST(InterleavedRangeTest, VectorIntEmpty) { + SmallVector V = {}; + EXPECT_EQ("", interleaved(V).str()); + EXPECT_EQ("{{}}", interleaved(V, ",", "{{", "}}").str()); + EXPECT_EQ("[]", interleaved_array(V).str()); + EXPECT_EQ("", interleaved(V, ";").str()); +} + +TEST(InterleavedRangeTest, VectorIntOneElem) { + SmallVector V = {42}; + EXPECT_EQ("42", interleaved(V).str()); + EXPECT_EQ("{{42}}", interleaved(V, ",", "{{", "}}").str()); + EXPECT_EQ("[42]", interleaved_array(V).str()); + EXPECT_EQ("42", interleaved(V, ";").str()); +} + +struct CustomPrint { + int N; + friend raw_ostream &operator<<(raw_ostream &OS, const CustomPrint &CP) { + OS << "$$" << CP.N << "##"; + return OS; + } +}; + +TEST(InterleavedRangeTest, CustomPrint) { + CustomPrint V[] = {{3}, {4}, {5}}; + EXPECT_EQ("$$3##, $$4##, $$5##", interleaved(V).str()); + EXPECT_EQ("{{$$3##;$$4##;$$5##}}", interleaved(V, ";", "{{", "}}").str()); + EXPECT_EQ("[$$3##, $$4##, $$5##]", interleaved_array(V).str()); +} + +} // namespace From 04b87e15e40f8857e29ade8321b8b67691545a50 Mon Sep 17 00:00:00 2001 From: Kareem Ergawy Date: Wed, 16 Apr 2025 06:14:38 +0200 Subject: [PATCH 62/63] [flang][fir] Lower `do concurrent` loop nests to `fir.do_concurrent` (#132904) Adds support for lowering `do concurrent` nests from PFT to the new `fir.do_concurrent` MLIR op as well as its special terminator `fir.do_concurrent.loop` which models the actual loop nest. To that end, this PR emits the allocations for the iteration variables within the block of the `fir.do_concurrent` op and creates a region for the `fir.do_concurrent.loop` op that accepts arguments equal in number to the number of the input `do concurrent` iteration ranges. For example, given the following input: ```fortran do concurrent(i=1:10, j=11:20) end do ``` the changes in this PR emit the following MLIR: ```mlir fir.do_concurrent { %22 = fir.alloca i32 {bindc_name = "i"} %23:2 = hlfir.declare %22 {uniq_name = "_QFsub1Ei"} : (!fir.ref) -> (!fir.ref, !fir.ref) %24 = fir.alloca i32 {bindc_name = "j"} %25:2 = hlfir.declare %24 {uniq_name = "_QFsub1Ej"} : (!fir.ref) -> (!fir.ref, !fir.ref) fir.do_concurrent.loop (%arg1, %arg2) = (%18, %20) to (%19, %21) step (%c1, %c1_0) { %26 = fir.convert %arg1 : (index) -> i32 fir.store %26 to %23#0 : !fir.ref %27 = fir.convert %arg2 : (index) -> i32 fir.store %27 to %25#0 : !fir.ref } } ``` --- flang/lib/Lower/Bridge.cpp | 228 +++++++++++------- flang/lib/Optimizer/Builder/FIRBuilder.cpp | 3 + flang/test/Lower/do_concurrent.f90 | 39 ++- .../do_concurrent_local_default_init.f90 | 4 +- flang/test/Lower/loops.f90 | 37 +-- flang/test/Lower/loops3.f90 | 4 +- flang/test/Lower/nsw.f90 | 5 +- .../Transforms/DoConcurrent/basic_host.f90 | 3 + .../DoConcurrent/locally_destroyed_temp.f90 | 3 + .../DoConcurrent/loop_nest_test.f90 | 3 + .../multiple_iteration_ranges.f90 | 3 + .../DoConcurrent/non_const_bounds.f90 | 3 + .../DoConcurrent/not_perfectly_nested.f90 | 3 + 13 files changed, 208 insertions(+), 130 deletions(-) diff --git a/flang/lib/Lower/Bridge.cpp b/flang/lib/Lower/Bridge.cpp index b4d1197822a43..625dd116fe726 100644 --- a/flang/lib/Lower/Bridge.cpp +++ b/flang/lib/Lower/Bridge.cpp @@ -94,10 +94,11 @@ struct IncrementLoopInfo { template explicit IncrementLoopInfo(Fortran::semantics::Symbol &sym, const T &lower, const T &upper, const std::optional &step, - bool isUnordered = false) + bool isConcurrent = false) : loopVariableSym{&sym}, lowerExpr{Fortran::semantics::GetExpr(lower)}, upperExpr{Fortran::semantics::GetExpr(upper)}, - stepExpr{Fortran::semantics::GetExpr(step)}, isUnordered{isUnordered} {} + stepExpr{Fortran::semantics::GetExpr(step)}, + isConcurrent{isConcurrent} {} IncrementLoopInfo(IncrementLoopInfo &&) = default; IncrementLoopInfo &operator=(IncrementLoopInfo &&x) = default; @@ -120,7 +121,7 @@ struct IncrementLoopInfo { const Fortran::lower::SomeExpr *upperExpr; const Fortran::lower::SomeExpr *stepExpr; const Fortran::lower::SomeExpr *maskExpr = nullptr; - bool isUnordered; // do concurrent, forall + bool isConcurrent; llvm::SmallVector localSymList; llvm::SmallVector localInitSymList; llvm::SmallVector< @@ -130,7 +131,7 @@ struct IncrementLoopInfo { mlir::Value loopVariable = nullptr; // Data members for structured loops. - fir::DoLoopOp doLoop = nullptr; + mlir::Operation *loopOp = nullptr; // Data members for unstructured loops. bool hasRealControl = false; @@ -1980,7 +1981,7 @@ class FirConverter : public Fortran::lower::AbstractConverter { llvm_unreachable("illegal reduction operator"); } - /// Collect DO CONCURRENT or FORALL loop control information. + /// Collect DO CONCURRENT loop control information. IncrementLoopNestInfo getConcurrentControl( const Fortran::parser::ConcurrentHeader &header, const std::list &localityList = {}) { @@ -2291,8 +2292,14 @@ class FirConverter : public Fortran::lower::AbstractConverter { mlir::LLVM::LoopAnnotationAttr la = mlir::LLVM::LoopAnnotationAttr::get( builder->getContext(), {}, /*vectorize=*/va, {}, /*unroll*/ ua, /*unroll_and_jam*/ uja, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}); - if (has_attrs) - info.doLoop.setLoopAnnotationAttr(la); + if (has_attrs) { + if (auto loopOp = mlir::dyn_cast(info.loopOp)) + loopOp.setLoopAnnotationAttr(la); + + if (auto doConcurrentOp = + mlir::dyn_cast(info.loopOp)) + doConcurrentOp.setLoopAnnotationAttr(la); + } } /// Generate FIR to begin a structured or unstructured increment loop nest. @@ -2301,96 +2308,77 @@ class FirConverter : public Fortran::lower::AbstractConverter { llvm::SmallVectorImpl &dirs) { assert(!incrementLoopNestInfo.empty() && "empty loop nest"); mlir::Location loc = toLocation(); - mlir::Operation *boundsAndStepIP = nullptr; mlir::arith::IntegerOverflowFlags iofBackup{}; + llvm::SmallVector nestLBs; + llvm::SmallVector nestUBs; + llvm::SmallVector nestSts; + llvm::SmallVector nestReduceOperands; + llvm::SmallVector nestReduceAttrs; + bool genDoConcurrent = false; + for (IncrementLoopInfo &info : incrementLoopNestInfo) { - mlir::Value lowerValue; - mlir::Value upperValue; - mlir::Value stepValue; + genDoConcurrent = info.isStructured() && info.isConcurrent; - { - mlir::OpBuilder::InsertionGuard guard(*builder); + if (!genDoConcurrent) + info.loopVariable = genLoopVariableAddress(loc, *info.loopVariableSym, + info.isConcurrent); - // Set the IP before the first loop in the nest so that all nest bounds - // and step values are created outside the nest. - if (boundsAndStepIP) - builder->setInsertionPointAfter(boundsAndStepIP); + if (!getLoweringOptions().getIntegerWrapAround()) { + iofBackup = builder->getIntegerOverflowFlags(); + builder->setIntegerOverflowFlags( + mlir::arith::IntegerOverflowFlags::nsw); + } - info.loopVariable = genLoopVariableAddress(loc, *info.loopVariableSym, - info.isUnordered); - if (!getLoweringOptions().getIntegerWrapAround()) { - iofBackup = builder->getIntegerOverflowFlags(); - builder->setIntegerOverflowFlags( - mlir::arith::IntegerOverflowFlags::nsw); - } - lowerValue = genControlValue(info.lowerExpr, info); - upperValue = genControlValue(info.upperExpr, info); - bool isConst = true; - stepValue = genControlValue(info.stepExpr, info, - info.isStructured() ? nullptr : &isConst); - if (!getLoweringOptions().getIntegerWrapAround()) - builder->setIntegerOverflowFlags(iofBackup); - boundsAndStepIP = stepValue.getDefiningOp(); - - // Use a temp variable for unstructured loops with non-const step. - if (!isConst) { - info.stepVariable = - builder->createTemporary(loc, stepValue.getType()); - boundsAndStepIP = - builder->create(loc, stepValue, info.stepVariable); + nestLBs.push_back(genControlValue(info.lowerExpr, info)); + nestUBs.push_back(genControlValue(info.upperExpr, info)); + bool isConst = true; + nestSts.push_back(genControlValue( + info.stepExpr, info, info.isStructured() ? nullptr : &isConst)); + + if (!getLoweringOptions().getIntegerWrapAround()) + builder->setIntegerOverflowFlags(iofBackup); + + // Use a temp variable for unstructured loops with non-const step. + if (!isConst) { + mlir::Value stepValue = nestSts.back(); + info.stepVariable = builder->createTemporary(loc, stepValue.getType()); + builder->create(loc, stepValue, info.stepVariable); + } + + if (genDoConcurrent && nestReduceOperands.empty()) { + // Create DO CONCURRENT reduce operands and attributes + for (const auto &reduceSym : info.reduceSymList) { + const fir::ReduceOperationEnum reduceOperation = reduceSym.first; + const Fortran::semantics::Symbol *sym = reduceSym.second; + fir::ExtendedValue exv = getSymbolExtendedValue(*sym, nullptr); + nestReduceOperands.push_back(fir::getBase(exv)); + auto reduceAttr = + fir::ReduceAttr::get(builder->getContext(), reduceOperation); + nestReduceAttrs.push_back(reduceAttr); } } + } + for (auto [info, lowerValue, upperValue, stepValue] : + llvm::zip_equal(incrementLoopNestInfo, nestLBs, nestUBs, nestSts)) { // Structured loop - generate fir.do_loop. if (info.isStructured()) { + if (genDoConcurrent) + continue; + + // The loop variable is a doLoop op argument. mlir::Type loopVarType = info.getLoopVariableType(); - mlir::Value loopValue; - if (info.isUnordered) { - llvm::SmallVector reduceOperands; - llvm::SmallVector reduceAttrs; - // Create DO CONCURRENT reduce operands and attributes - for (const auto &reduceSym : info.reduceSymList) { - const fir::ReduceOperationEnum reduce_operation = reduceSym.first; - const Fortran::semantics::Symbol *sym = reduceSym.second; - fir::ExtendedValue exv = getSymbolExtendedValue(*sym, nullptr); - reduceOperands.push_back(fir::getBase(exv)); - auto reduce_attr = - fir::ReduceAttr::get(builder->getContext(), reduce_operation); - reduceAttrs.push_back(reduce_attr); - } - // The loop variable value is explicitly updated. - info.doLoop = builder->create( - loc, lowerValue, upperValue, stepValue, /*unordered=*/true, - /*finalCountValue=*/false, /*iterArgs=*/std::nullopt, - llvm::ArrayRef(reduceOperands), reduceAttrs); - builder->setInsertionPointToStart(info.doLoop.getBody()); - loopValue = builder->createConvert(loc, loopVarType, - info.doLoop.getInductionVar()); - } else { - // The loop variable is a doLoop op argument. - info.doLoop = builder->create( - loc, lowerValue, upperValue, stepValue, /*unordered=*/false, - /*finalCountValue=*/true, - builder->createConvert(loc, loopVarType, lowerValue)); - builder->setInsertionPointToStart(info.doLoop.getBody()); - loopValue = info.doLoop.getRegionIterArgs()[0]; - } + auto loopOp = builder->create( + loc, lowerValue, upperValue, stepValue, /*unordered=*/false, + /*finalCountValue=*/true, + builder->createConvert(loc, loopVarType, lowerValue)); + info.loopOp = loopOp; + builder->setInsertionPointToStart(loopOp.getBody()); + mlir::Value loopValue = loopOp.getRegionIterArgs()[0]; + // Update the loop variable value in case it has non-index references. builder->create(loc, loopValue, info.loopVariable); - if (info.maskExpr) { - Fortran::lower::StatementContext stmtCtx; - mlir::Value maskCond = createFIRExpr(loc, info.maskExpr, stmtCtx); - stmtCtx.finalizeAndReset(); - mlir::Value maskCondCast = - builder->createConvert(loc, builder->getI1Type(), maskCond); - auto ifOp = builder->create(loc, maskCondCast, - /*withElseRegion=*/false); - builder->setInsertionPointToStart(&ifOp.getThenRegion().front()); - } - if (info.hasLocalitySpecs()) - handleLocalitySpecs(info); - addLoopAnnotationAttr(info, dirs); continue; } @@ -2454,6 +2442,60 @@ class FirConverter : public Fortran::lower::AbstractConverter { builder->restoreInsertionPoint(insertPt); } } + + if (genDoConcurrent) { + auto loopWrapperOp = builder->create(loc); + builder->setInsertionPointToStart( + builder->createBlock(&loopWrapperOp.getRegion())); + + for (IncrementLoopInfo &info : llvm::reverse(incrementLoopNestInfo)) { + info.loopVariable = genLoopVariableAddress(loc, *info.loopVariableSym, + info.isConcurrent); + } + + builder->setInsertionPointToEnd(loopWrapperOp.getBody()); + auto loopOp = builder->create( + loc, nestLBs, nestUBs, nestSts, nestReduceOperands, + nestReduceAttrs.empty() + ? nullptr + : mlir::ArrayAttr::get(builder->getContext(), nestReduceAttrs), + nullptr); + + llvm::SmallVector loopBlockArgTypes( + incrementLoopNestInfo.size(), builder->getIndexType()); + llvm::SmallVector loopBlockArgLocs( + incrementLoopNestInfo.size(), loc); + mlir::Region &loopRegion = loopOp.getRegion(); + mlir::Block *loopBlock = builder->createBlock( + &loopRegion, loopRegion.begin(), loopBlockArgTypes, loopBlockArgLocs); + builder->setInsertionPointToStart(loopBlock); + + for (auto [info, blockArg] : + llvm::zip_equal(incrementLoopNestInfo, loopBlock->getArguments())) { + info.loopOp = loopOp; + mlir::Value loopValue = + builder->createConvert(loc, info.getLoopVariableType(), blockArg); + builder->create(loc, loopValue, info.loopVariable); + + if (info.maskExpr) { + Fortran::lower::StatementContext stmtCtx; + mlir::Value maskCond = createFIRExpr(loc, info.maskExpr, stmtCtx); + stmtCtx.finalizeAndReset(); + mlir::Value maskCondCast = + builder->createConvert(loc, builder->getI1Type(), maskCond); + auto ifOp = builder->create(loc, maskCondCast, + /*withElseRegion=*/false); + builder->setInsertionPointToStart(&ifOp.getThenRegion().front()); + } + } + + IncrementLoopInfo &innermostInfo = incrementLoopNestInfo.back(); + + if (innermostInfo.hasLocalitySpecs()) + handleLocalitySpecs(innermostInfo); + + addLoopAnnotationAttr(innermostInfo, dirs); + } } /// Generate FIR to end a structured or unstructured increment loop nest. @@ -2470,29 +2512,31 @@ class FirConverter : public Fortran::lower::AbstractConverter { it != rend; ++it) { IncrementLoopInfo &info = *it; if (info.isStructured()) { - // End fir.do_loop. - if (info.isUnordered) { - builder->setInsertionPointAfter(info.doLoop); + // End fir.do_concurent.loop. + if (info.isConcurrent) { + builder->setInsertionPointAfter(info.loopOp->getParentOp()); continue; } + + // End fir.do_loop. // Decrement tripVariable. - builder->setInsertionPointToEnd(info.doLoop.getBody()); + auto doLoopOp = mlir::cast(info.loopOp); + builder->setInsertionPointToEnd(doLoopOp.getBody()); llvm::SmallVector results; results.push_back(builder->create( - loc, info.doLoop.getInductionVar(), info.doLoop.getStep(), - iofAttr)); + loc, doLoopOp.getInductionVar(), doLoopOp.getStep(), iofAttr)); // Step loopVariable to help optimizations such as vectorization. // Induction variable elimination will clean up as necessary. mlir::Value step = builder->createConvert( - loc, info.getLoopVariableType(), info.doLoop.getStep()); + loc, info.getLoopVariableType(), doLoopOp.getStep()); mlir::Value loopVar = builder->create(loc, info.loopVariable); results.push_back( builder->create(loc, loopVar, step, iofAttr)); builder->create(loc, results); - builder->setInsertionPointAfter(info.doLoop); + builder->setInsertionPointAfter(doLoopOp); // The loop control variable may be used after the loop. - builder->create(loc, info.doLoop.getResult(1), + builder->create(loc, doLoopOp.getResult(1), info.loopVariable); continue; } diff --git a/flang/lib/Optimizer/Builder/FIRBuilder.cpp b/flang/lib/Optimizer/Builder/FIRBuilder.cpp index 3cf9b5ae72d9e..d35367d7657cf 100644 --- a/flang/lib/Optimizer/Builder/FIRBuilder.cpp +++ b/flang/lib/Optimizer/Builder/FIRBuilder.cpp @@ -280,6 +280,9 @@ mlir::Block *fir::FirOpBuilder::getAllocaBlock() { if (auto cufKernelOp = getRegion().getParentOfType()) return &cufKernelOp.getRegion().front(); + if (auto doConcurentOp = getRegion().getParentOfType()) + return doConcurentOp.getBody(); + return getEntryBlock(); } diff --git a/flang/test/Lower/do_concurrent.f90 b/flang/test/Lower/do_concurrent.f90 index ef93d2d6b035b..cc113f59c35e3 100644 --- a/flang/test/Lower/do_concurrent.f90 +++ b/flang/test/Lower/do_concurrent.f90 @@ -14,6 +14,9 @@ subroutine sub1(n) implicit none integer :: n, m, i, j, k integer, dimension(n) :: a +!CHECK: %[[N_DECL:.*]]:2 = hlfir.declare %{{.*}} dummy_scope %{{.*}} {uniq_name = "_QFsub1En"} +!CHECK: %[[A_DECL:.*]]:2 = hlfir.declare %{{.*}}(%{{.*}}) {uniq_name = "_QFsub1Ea"} + !CHECK: %[[LB1:.*]] = arith.constant 1 : i32 !CHECK: %[[LB1_CVT:.*]] = fir.convert %[[LB1]] : (i32) -> index !CHECK: %[[UB1:.*]] = fir.load %{{.*}}#0 : !fir.ref @@ -29,10 +32,30 @@ subroutine sub1(n) !CHECK: %[[UB3:.*]] = arith.constant 10 : i32 !CHECK: %[[UB3_CVT:.*]] = fir.convert %[[UB3]] : (i32) -> index -!CHECK: fir.do_loop %{{.*}} = %[[LB1_CVT]] to %[[UB1_CVT]] step %{{.*}} unordered -!CHECK: fir.do_loop %{{.*}} = %[[LB2_CVT]] to %[[UB2_CVT]] step %{{.*}} unordered -!CHECK: fir.do_loop %{{.*}} = %[[LB3_CVT]] to %[[UB3_CVT]] step %{{.*}} unordered +!CHECK: fir.do_concurrent +!CHECK: %[[I:.*]] = fir.alloca i32 {bindc_name = "i"} +!CHECK: %[[I_DECL:.*]]:2 = hlfir.declare %[[I]] +!CHECK: %[[J:.*]] = fir.alloca i32 {bindc_name = "j"} +!CHECK: %[[J_DECL:.*]]:2 = hlfir.declare %[[J]] +!CHECK: %[[K:.*]] = fir.alloca i32 {bindc_name = "k"} +!CHECK: %[[K_DECL:.*]]:2 = hlfir.declare %[[K]] + +!CHECK: fir.do_concurrent.loop (%[[I_IV:.*]], %[[J_IV:.*]], %[[K_IV:.*]]) = +!CHECK-SAME: (%[[LB1_CVT]], %[[LB2_CVT]], %[[LB3_CVT]]) to +!CHECK-SAME: (%[[UB1_CVT]], %[[UB2_CVT]], %[[UB3_CVT]]) step +!CHECK-SAME: (%{{.*}}, %{{.*}}, %{{.*}}) { +!CHECK: %[[I_IV_CVT:.*]] = fir.convert %[[I_IV]] : (index) -> i32 +!CHECK: fir.store %[[I_IV_CVT]] to %[[I_DECL]]#0 : !fir.ref +!CHECK: %[[J_IV_CVT:.*]] = fir.convert %[[J_IV]] : (index) -> i32 +!CHECK: fir.store %[[J_IV_CVT]] to %[[J_DECL]]#0 : !fir.ref +!CHECK: %[[K_IV_CVT:.*]] = fir.convert %[[K_IV]] : (index) -> i32 +!CHECK: fir.store %[[K_IV_CVT]] to %[[K_DECL]]#0 : !fir.ref +!CHECK: %[[N_VAL:.*]] = fir.load %[[N_DECL]]#0 : !fir.ref +!CHECK: %[[I_VAL:.*]] = fir.load %[[I_DECL]]#0 : !fir.ref +!CHECK: %[[I_VAL_CVT:.*]] = fir.convert %[[I_VAL]] : (i32) -> i64 +!CHECK: %[[A_ELEM:.*]] = hlfir.designate %[[A_DECL]]#0 (%[[I_VAL_CVT]]) +!CHECK: hlfir.assign %[[N_VAL]] to %[[A_ELEM]] : i32, !fir.ref do concurrent(i=1:n, j=1:bar(n*m, n/m), k=5:10) a(i) = n end do @@ -45,14 +68,17 @@ subroutine sub2(n) integer, dimension(n) :: a !CHECK: %[[LB1:.*]] = arith.constant 1 : i32 !CHECK: %[[LB1_CVT:.*]] = fir.convert %[[LB1]] : (i32) -> index -!CHECK: %[[UB1:.*]] = fir.load %5#0 : !fir.ref +!CHECK: %[[UB1:.*]] = fir.load %{{.*}}#0 : !fir.ref !CHECK: %[[UB1_CVT:.*]] = fir.convert %[[UB1]] : (i32) -> index -!CHECK: fir.do_loop %{{.*}} = %[[LB1_CVT]] to %[[UB1_CVT]] step %{{.*}} unordered +!CHECK: fir.do_concurrent +!CHECK: fir.do_concurrent.loop (%{{.*}}) = (%[[LB1_CVT]]) to (%[[UB1_CVT]]) step (%{{.*}}) + !CHECK: %[[LB2:.*]] = arith.constant 1 : i32 !CHECK: %[[LB2_CVT:.*]] = fir.convert %[[LB2]] : (i32) -> index !CHECK: %[[UB2:.*]] = fir.call @_QPbar(%{{.*}}, %{{.*}}) proc_attrs fastmath : (!fir.ref, !fir.ref) -> i32 !CHECK: %[[UB2_CVT:.*]] = fir.convert %[[UB2]] : (i32) -> index -!CHECK: fir.do_loop %{{.*}} = %[[LB2_CVT]] to %[[UB2_CVT]] step %{{.*}} unordered +!CHECK: fir.do_concurrent +!CHECK: fir.do_concurrent.loop (%{{.*}}) = (%[[LB2_CVT]]) to (%[[UB2_CVT]]) step (%{{.*}}) do concurrent(i=1:n) do concurrent(j=1:bar(n*m, n/m)) a(i) = n @@ -60,7 +86,6 @@ subroutine sub2(n) end do end subroutine - !CHECK-LABEL: unstructured subroutine unstructured(inner_step) integer(4) :: i, j, inner_step diff --git a/flang/test/Lower/do_concurrent_local_default_init.f90 b/flang/test/Lower/do_concurrent_local_default_init.f90 index 7652e4fcd0402..207704ac1a990 100644 --- a/flang/test/Lower/do_concurrent_local_default_init.f90 +++ b/flang/test/Lower/do_concurrent_local_default_init.f90 @@ -29,7 +29,7 @@ subroutine test_default_init() ! CHECK-SAME: %[[VAL_0:.*]]: !fir.ref>>>> {fir.bindc_name = "p"}) { ! CHECK: %[[VAL_6:.*]] = fir.load %[[VAL_0]] : !fir.ref>>>> ! CHECK: %[[VAL_7:.*]] = fir.box_elesize %[[VAL_6]] : (!fir.box>>>) -> index -! CHECK: fir.do_loop +! CHECK: fir.do_concurrent.loop ! CHECK: %[[VAL_16:.*]] = fir.alloca !fir.box>>> {bindc_name = "p", pinned, uniq_name = "_QFtest_ptrEp"} ! CHECK: %[[VAL_17:.*]] = fir.zero_bits !fir.ptr>> ! CHECK: %[[VAL_18:.*]] = arith.constant 0 : index @@ -43,7 +43,7 @@ subroutine test_default_init() ! CHECK: } ! CHECK-LABEL: func.func @_QPtest_default_init( -! CHECK: fir.do_loop +! CHECK: fir.do_concurrent.loop ! CHECK: %[[VAL_26:.*]] = fir.alloca !fir.type<_QFtest_default_initTt{i:i32}> {bindc_name = "a", pinned, uniq_name = "_QFtest_default_initEa"} ! CHECK: %[[VAL_27:.*]] = fir.embox %[[VAL_26]] : (!fir.ref>) -> !fir.box> ! CHECK: %[[VAL_30:.*]] = fir.convert %[[VAL_27]] : (!fir.box>) -> !fir.box diff --git a/flang/test/Lower/loops.f90 b/flang/test/Lower/loops.f90 index ea65ba3e4d66d..60df27a591dc3 100644 --- a/flang/test/Lower/loops.f90 +++ b/flang/test/Lower/loops.f90 @@ -2,15 +2,6 @@ ! CHECK-LABEL: loop_test subroutine loop_test - ! CHECK: %[[VAL_2:.*]] = fir.alloca i16 {bindc_name = "i"} - ! CHECK: %[[VAL_3:.*]] = fir.alloca i16 {bindc_name = "i"} - ! CHECK: %[[VAL_4:.*]] = fir.alloca i16 {bindc_name = "i"} - ! CHECK: %[[VAL_5:.*]] = fir.alloca i8 {bindc_name = "k"} - ! CHECK: %[[VAL_6:.*]] = fir.alloca i8 {bindc_name = "j"} - ! CHECK: %[[VAL_7:.*]] = fir.alloca i8 {bindc_name = "i"} - ! CHECK: %[[VAL_8:.*]] = fir.alloca i32 {bindc_name = "k"} - ! CHECK: %[[VAL_9:.*]] = fir.alloca i32 {bindc_name = "j"} - ! CHECK: %[[VAL_10:.*]] = fir.alloca i32 {bindc_name = "i"} ! CHECK: %[[VAL_11:.*]] = fir.alloca !fir.array<5x5x5xi32> {bindc_name = "a", uniq_name = "_QFloop_testEa"} ! CHECK: %[[VAL_12:.*]] = fir.alloca i32 {bindc_name = "asum", uniq_name = "_QFloop_testEasum"} ! CHECK: %[[VAL_13:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFloop_testEi"} @@ -25,7 +16,7 @@ subroutine loop_test j = 200 k = 300 - ! CHECK-COUNT-3: fir.do_loop {{.*}} unordered + ! CHECK: fir.do_concurrent.loop (%{{.*}}, %{{.*}}, %{{.*}}) = {{.*}} do concurrent (i=1:5, j=1:5, k=1:5) ! shared(a) ! CHECK: fir.coordinate_of a(i,j,k) = 0 @@ -33,7 +24,7 @@ subroutine loop_test ! CHECK: fir.call @_FortranAioBeginExternalListOutput print*, 'A:', i, j, k - ! CHECK-COUNT-3: fir.do_loop {{.*}} unordered + ! CHECK: fir.do_concurrent.loop (%{{.*}}, %{{.*}}, %{{.*}}) = {{.*}} ! CHECK: fir.if do concurrent (integer(1)::i=1:5, j=1:5, k=1:5, i.ne.j .and. k.ne.3) shared(a) ! CHECK-COUNT-2: fir.coordinate_of @@ -53,7 +44,7 @@ subroutine loop_test ! CHECK: fir.call @_FortranAioBeginExternalListOutput print*, 'B:', i, j, k, '-', asum - ! CHECK: fir.do_loop {{.*}} unordered + ! CHECK: fir.do_concurrent.loop (%{{.*}}) = {{.*}} ! CHECK-COUNT-2: fir.if do concurrent (integer(2)::i=1:5, i.ne.3) if (i.eq.2 .or. i.eq.4) goto 5 ! fir.if @@ -62,7 +53,7 @@ subroutine loop_test 5 continue enddo - ! CHECK: fir.do_loop {{.*}} unordered + ! CHECK: fir.do_concurrent.loop (%{{.*}}) = {{.*}} ! CHECK-COUNT-2: fir.if do concurrent (integer(2)::i=1:5, i.ne.3) if (i.eq.2 .or. i.eq.4) then ! fir.if @@ -93,10 +84,6 @@ end subroutine loop_test ! CHECK-LABEL: c.func @_QPlis subroutine lis(n) - ! CHECK-DAG: fir.alloca i32 {bindc_name = "m"} - ! CHECK-DAG: fir.alloca i32 {bindc_name = "j"} - ! CHECK-DAG: fir.alloca i32 {bindc_name = "i"} - ! CHECK-DAG: fir.alloca i8 {bindc_name = "i"} ! CHECK-DAG: fir.alloca i32 {bindc_name = "j", uniq_name = "_QFlisEj"} ! CHECK-DAG: fir.alloca i32 {bindc_name = "k", uniq_name = "_QFlisEk"} ! CHECK-DAG: fir.alloca !fir.box>> {bindc_name = "p", uniq_name = "_QFlisEp"} @@ -117,8 +104,8 @@ subroutine lis(n) ! CHECK: } r = 0 - ! CHECK: fir.do_loop %arg1 = %{{.*}} to %{{.*}} step %{{.*}} unordered { - ! CHECK: fir.do_loop %arg2 = %{{.*}} to %{{.*}} step %c1{{.*}} iter_args(%arg3 = %{{.*}}) -> (index, i32) { + ! CHECK: fir.do_concurrent { + ! CHECK: fir.do_concurrent.loop (%{{.*}}) = (%{{.*}}) to (%{{.*}}) step (%{{.*}}) { ! CHECK: } ! CHECK: } do concurrent (integer(kind=1)::i=n:1:-1) @@ -128,16 +115,18 @@ subroutine lis(n) enddo enddo - ! CHECK: fir.do_loop %arg1 = %{{.*}} to %{{.*}} step %c1{{.*}} unordered { - ! CHECK: fir.do_loop %arg2 = %{{.*}} to %{{.*}} step %c1{{.*}} unordered { + ! CHECK: fir.do_concurrent.loop (%{{.*}}, %{{.*}}) = (%{{.*}}, %{{.*}}) to (%{{.*}}, %{{.*}}) step (%{{.*}}, %{{.*}}) { ! CHECK: fir.if %{{.*}} { ! CHECK: %[[V_95:[0-9]+]] = fir.alloca !fir.array, %{{.*}}, %{{.*}} {bindc_name = "t", pinned, uniq_name = "_QFlisEt"} ! CHECK: %[[V_96:[0-9]+]] = fir.alloca !fir.box>> {bindc_name = "p", pinned, uniq_name = "_QFlisEp"} ! CHECK: fir.store %{{.*}} to %[[V_96]] : !fir.ref>>> ! CHECK: fir.do_loop %arg3 = %{{.*}} to %{{.*}} step %c1{{.*}} iter_args(%arg4 = %{{.*}}) -> (index, i32) { - ! CHECK: fir.do_loop %arg5 = %{{.*}} to %{{.*}} step %c1{{.*}} unordered { - ! CHECK: fir.load %[[V_96]] : !fir.ref>>> - ! CHECK: fir.convert %[[V_95]] : (!fir.ref>) -> !fir.ref> + ! CHECK: fir.do_concurrent { + ! CHECK: fir.alloca i32 {bindc_name = "m"} + ! CHECK: fir.do_concurrent.loop (%{{.*}}) = (%{{.*}}) to (%{{.*}}) step (%{{.*}}) { + ! CHECK: fir.load %[[V_96]] : !fir.ref>>> + ! CHECK: fir.convert %[[V_95]] : (!fir.ref>) -> !fir.ref> + ! CHECK: } ! CHECK: } ! CHECK: } ! CHECK: fir.convert %[[V_95]] : (!fir.ref>) -> !fir.ref> diff --git a/flang/test/Lower/loops3.f90 b/flang/test/Lower/loops3.f90 index 78f39e1013082..84db1972cca16 100644 --- a/flang/test/Lower/loops3.f90 +++ b/flang/test/Lower/loops3.f90 @@ -12,9 +12,7 @@ subroutine loop_test ! CHECK: %[[VAL_0:.*]] = fir.alloca f32 {bindc_name = "m", uniq_name = "_QFloop_testEm"} ! CHECK: %[[VAL_1:.*]] = fir.address_of(@_QFloop_testEsum) : !fir.ref - ! CHECK: fir.do_loop %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} unordered reduce(#fir.reduce_attr -> %[[VAL_1:.*]] : !fir.ref, #fir.reduce_attr -> %[[VAL_0:.*]] : !fir.ref) { - ! CHECK: fir.do_loop %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} unordered reduce(#fir.reduce_attr -> %[[VAL_1:.*]] : !fir.ref, #fir.reduce_attr -> %[[VAL_0:.*]] : !fir.ref) { - ! CHECK: fir.do_loop %{{.*}} = %{{.*}} to %{{.*}} step %{{.*}} unordered reduce(#fir.reduce_attr -> %[[VAL_1:.*]] : !fir.ref, #fir.reduce_attr -> %[[VAL_0:.*]] : !fir.ref) { + ! CHECK: fir.do_concurrent.loop ({{.*}}) = ({{.*}}) to ({{.*}}) step ({{.*}}) reduce(#fir.reduce_attr -> %[[VAL_1:.*]] : !fir.ref, #fir.reduce_attr -> %[[VAL_0:.*]] : !fir.ref) { do concurrent (i=1:5, j=1:5, k=1:5) local(tmp) reduce(+:sum) reduce(max:m) tmp = i + j + k sum = tmp + sum diff --git a/flang/test/Lower/nsw.f90 b/flang/test/Lower/nsw.f90 index 4ee9e5da829e6..2ec1efb2af42a 100644 --- a/flang/test/Lower/nsw.f90 +++ b/flang/test/Lower/nsw.f90 @@ -139,7 +139,6 @@ subroutine loop_params3(a,lb,ub,st) ! CHECK-LABEL: func.func @_QPloop_params3( ! CHECK: %[[VAL_4:.*]] = arith.constant 2 : i32 ! CHECK: %[[VAL_5:.*]] = arith.constant 1 : i32 -! CHECK: %[[VAL_9:.*]] = fir.declare %{{.*}}i"} : (!fir.ref) -> !fir.ref ! CHECK: %[[VAL_11:.*]] = fir.declare %{{.*}}lb"} : (!fir.ref, !fir.dscope) -> !fir.ref ! CHECK: %[[VAL_12:.*]] = fir.declare %{{.*}}ub"} : (!fir.ref, !fir.dscope) -> !fir.ref ! CHECK: %[[VAL_14:.*]] = fir.declare %{{.*}}i"} : (!fir.ref) -> !fir.ref @@ -153,4 +152,6 @@ subroutine loop_params3(a,lb,ub,st) ! CHECK: %[[VAL_31:.*]] = fir.load %[[VAL_15]] : !fir.ref ! CHECK: %[[VAL_32:.*]] = arith.muli %[[VAL_31]], %[[VAL_4]] overflow : i32 ! CHECK: %[[VAL_33:.*]] = fir.convert %[[VAL_32]] : (i32) -> index -! CHECK: fir.do_loop %[[VAL_34:.*]] = %[[VAL_28]] to %[[VAL_30]] step %[[VAL_33]] unordered { +! CHECK: fir.do_concurrent { +! CHECK: %[[VAL_9:.*]] = fir.declare %{{.*}}i"} : (!fir.ref) -> !fir.ref +! CHECK: fir.do_concurrent.loop (%[[VAL_34:.*]]) = (%[[VAL_28]]) to (%[[VAL_30]]) step (%[[VAL_33]]) { diff --git a/flang/test/Transforms/DoConcurrent/basic_host.f90 b/flang/test/Transforms/DoConcurrent/basic_host.f90 index 12f63031cbaee..b84d4481ac766 100644 --- a/flang/test/Transforms/DoConcurrent/basic_host.f90 +++ b/flang/test/Transforms/DoConcurrent/basic_host.f90 @@ -1,3 +1,6 @@ +! Fails until we update the pass to use the `fir.do_concurrent` op. +! XFAIL: * + ! Tests mapping of a basic `do concurrent` loop to `!$omp parallel do`. ! RUN: %flang_fc1 -emit-hlfir -fopenmp -fdo-concurrent-to-openmp=host %s -o - \ diff --git a/flang/test/Transforms/DoConcurrent/locally_destroyed_temp.f90 b/flang/test/Transforms/DoConcurrent/locally_destroyed_temp.f90 index f82696669eca6..4e13c0919589a 100644 --- a/flang/test/Transforms/DoConcurrent/locally_destroyed_temp.f90 +++ b/flang/test/Transforms/DoConcurrent/locally_destroyed_temp.f90 @@ -1,3 +1,6 @@ +! Fails until we update the pass to use the `fir.do_concurrent` op. +! XFAIL: * + ! Tests that "loop-local values" are properly handled by localizing them to the ! body of the loop nest. See `collectLoopLocalValues` and `localizeLoopLocalValue` ! for a definition of "loop-local values" and how they are handled. diff --git a/flang/test/Transforms/DoConcurrent/loop_nest_test.f90 b/flang/test/Transforms/DoConcurrent/loop_nest_test.f90 index 32bed61fe69e4..adc4a488d1ec9 100644 --- a/flang/test/Transforms/DoConcurrent/loop_nest_test.f90 +++ b/flang/test/Transforms/DoConcurrent/loop_nest_test.f90 @@ -1,3 +1,6 @@ +! Fails until we update the pass to use the `fir.do_concurrent` op. +! XFAIL: * + ! Tests loop-nest detection algorithm for do-concurrent mapping. ! REQUIRES: asserts diff --git a/flang/test/Transforms/DoConcurrent/multiple_iteration_ranges.f90 b/flang/test/Transforms/DoConcurrent/multiple_iteration_ranges.f90 index d0210726de83e..26800678d381c 100644 --- a/flang/test/Transforms/DoConcurrent/multiple_iteration_ranges.f90 +++ b/flang/test/Transforms/DoConcurrent/multiple_iteration_ranges.f90 @@ -1,3 +1,6 @@ +! Fails until we update the pass to use the `fir.do_concurrent` op. +! XFAIL: * + ! Tests mapping of a `do concurrent` loop with multiple iteration ranges. ! RUN: split-file %s %t diff --git a/flang/test/Transforms/DoConcurrent/non_const_bounds.f90 b/flang/test/Transforms/DoConcurrent/non_const_bounds.f90 index cd1bd4f98a3f5..23a3aae976c07 100644 --- a/flang/test/Transforms/DoConcurrent/non_const_bounds.f90 +++ b/flang/test/Transforms/DoConcurrent/non_const_bounds.f90 @@ -1,3 +1,6 @@ +! Fails until we update the pass to use the `fir.do_concurrent` op. +! XFAIL: * + ! RUN: %flang_fc1 -emit-hlfir -fopenmp -fdo-concurrent-to-openmp=host %s -o - \ ! RUN: | FileCheck %s diff --git a/flang/test/Transforms/DoConcurrent/not_perfectly_nested.f90 b/flang/test/Transforms/DoConcurrent/not_perfectly_nested.f90 index 184fdfe00d397..d1c02101318ab 100644 --- a/flang/test/Transforms/DoConcurrent/not_perfectly_nested.f90 +++ b/flang/test/Transforms/DoConcurrent/not_perfectly_nested.f90 @@ -1,3 +1,6 @@ +! Fails until we update the pass to use the `fir.do_concurrent` op. +! XFAIL: * + ! Tests that if `do concurrent` is not perfectly nested in its parent loop, that ! we skip converting the not-perfectly nested `do concurrent` loop. From 8af1dd336bae815197726cdf4b56ab5c60a3e888 Mon Sep 17 00:00:00 2001 From: Yang Zaizhou <91008302+Mxfg-incense@users.noreply.github.com> Date: Wed, 16 Apr 2025 01:32:50 +0800 Subject: [PATCH 63/63] [flang][docs] Fix typo in array description --- flang/docs/FortranForCProgrammers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flang/docs/FortranForCProgrammers.md b/flang/docs/FortranForCProgrammers.md index 50c83ed7e9bfe..135e6b71711d3 100644 --- a/flang/docs/FortranForCProgrammers.md +++ b/flang/docs/FortranForCProgrammers.md @@ -127,7 +127,7 @@ where type is not necessary. Arrays are not types in Fortran. Being an array is a property of an object or function, not of a type. Unlike C, one cannot have an array of arrays or an array of pointers, -although can can have an array of a derived type that has arrays or +although one can have an array of a derived type that has arrays or pointers as components. Arrays are multidimensional, and the number of dimensions is called the _rank_ of the array.