PiperOrigin-RevId: 698049836

…e ALL broadcast-like inputs on TFLite ops that support implicit broadcasting

PiperOrigin-RevId: 698054216

* Change default QNN graph config to use HTP FP16 precision backend config, this is required to correctly compile FP32 OPs.
* Create 1-element 1D tensor out of scalar value, QNN OP always use ranked tensor type as input.

PiperOrigin-RevId: 698081261

… the .td definition.

PiperOrigin-RevId: 698082807

 * Add FC Op legalization and test data.
 * Add Select/Select_v2 Op legalization.
 * Mics cleanups.

PiperOrigin-RevId: 698094953

PiperOrigin-RevId: 698111562

…rializing any modules.

Also pulled the deserialization a little further up the stack and only do it if the input doesn't already have a full module op.

PiperOrigin-RevId: 698116466

PiperOrigin-RevId: 698121993

PiperOrigin-RevId: 698132925

PiperOrigin-RevId: 698133024

… Todo (resolved)

PiperOrigin-RevId: 698133747

This is to fix issue with gsutil which expects Python 3.5-3.11:
```
Error: gsutil requires Python version 2.7 or 3.5-3.11, but a different version is installed.
```
PiperOrigin-RevId: 698134102

PiperOrigin-RevId: 698137647

PiperOrigin-RevId: 698150097

PiperOrigin-RevId: 698163185

…. This Extend() call would also lead to a memory assignment issue since it wasn't accompanied by the necessary chunk commit requests. We also add a VerifyAllocations() function that uses a BufferIntervalTree to check for overlapping Allocations before scheduling the asynchronous copies. This is an extra check for the correctness of MsaAlgorithm allocations, and is only applied if options_.verify is enabled in MSA options. options_.verify is disabled by default.

PiperOrigin-RevId: 698164396

PiperOrigin-RevId: 698164750

PiperOrigin-RevId: 698164921

This change adds the legalization pass from IFRT to VIFRT. Legalization uses a templated OpConversion class, which is refined via the `IFRT` <-> `VIFRT` and `mlir::Func::*` <-> `VIFRT` op mappings defined in `map_ifrt_to_vifrt.h` The change versions also `mlir::func::FuncOp`, `mlir::func::ReturnOp` and `mlir::func::CallOp` because this provides the following advantages: 1) we can use the templated OpConversion class rather than implementing a separate converter for each op, and 2) we can restrict the surface of possible breaking changes to just builtin types and attributes. Moreover, the change versions `mlir::FunctionType` and `mlir::TypeAttr` in order to be able to use the generic Op converter, and to restrict types allowed in functions (just builtin and IFRT types).

PiperOrigin-RevId: 698168526

Also fixed invalid C++ header usage.

PiperOrigin-RevId: 698170878

…td definition.

PiperOrigin-RevId: 698171237

PiperOrigin-RevId: 698174417

PiperOrigin-RevId: 698189797

PiperOrigin-RevId: 698196106

PiperOrigin-RevId: 698201598

PiperOrigin-RevId: 698212499

PiperOrigin-RevId: 698218778

PiperOrigin-RevId: 698221629

PiperOrigin-RevId: 698228306

… Todo(resolved)

PiperOrigin-RevId: 698230798

PiperOrigin-RevId: 698230884

PiperOrigin-RevId: 698237370

…remove the .td definition.

PiperOrigin-RevId: 698241447

… for bytes return from plugin in tests to avoid copy

PiperOrigin-RevId: 698251740

PiperOrigin-RevId: 698271808

PiperOrigin-RevId: 698294876

PiperOrigin-RevId: 698294898

PiperOrigin-RevId: 698297679

We have support for lowering PTX in the runtime, so we can just
use `MultiKernelLoaderSpec` and we get compilation and caching for free.

PiperOrigin-RevId: 698297929

PiperOrigin-RevId: 698302393

This is not needed since the runtime can compile PTX for us. Actually I'm surprised
that this even worked because this original code compiled PTX into CUBIN and then
forced the CUBIN into the PTX argument in the kernel creation helper. But this is
now all fixed.

PiperOrigin-RevId: 698303129

PiperOrigin-RevId: 698304950

…hunks

Also the usual drive-by cleanups:
- Remove unused includes
- Add explicit includes for things we depended on transitively
- Clean up dependencies of the build targets

PiperOrigin-RevId: 698317755

func-bufferize pass is removed by llvm/llvm-project@e394fec

PiperOrigin-RevId: 698322658

We turn on previously implemented heuristics by default.

PiperOrigin-RevId: 698324486

PiperOrigin-RevId: 698329980

Imported from GitHub PR openxla/xla#19363

Sets the loop iteration counter increment in the backward transformation of the collective pipeliner pass to account for cases with non-zero initial value of the loop iteration counter. See #16953 and #18568.
Copybara import of the project:

--
06137aa0618d372e2d4badbf16920bead9922cfb by Philipp Hack <[email protected]>:

Modifies the loop counter increment set in the backward transformation of the collective pipeliner.

--
6da45bcb26643d8994bf608f05230fa748286b02 by Philipp Hack <[email protected]>:

Modifies the loop counter increment set in the backward transformation of the collective pipeliner.

Merging this change closes #19363

PiperOrigin-RevId: 698342374

Dequantized static data is cached. However, when there are multiple subgraphs, the data is overwritten by each subgraph.

PiperOrigin-RevId: 698342673

…or fusion candidates.

Imported from GitHub PR openxla/xla#19393

Copybara import of the project:

--
ec107a12fbee6826f1f668218b7c7a40f5886420 by Ilia Sergachev <[email protected]>:

[GPU] Horizontal loop fusion: pass bitcasts when looking for fusion candidates.

--
71241097ce67412246ec18efca5165619601eace by Ilia Sergachev <[email protected]>:

simplify cuDNN norm test

Merging this change closes #19393

PiperOrigin-RevId: 698357838

PiperOrigin-RevId: 698360931

PiperOrigin-RevId: 698372450

…vice/gpu/te…

Imported from GitHub PR openxla/xla#19484

…sts:gpu_input_fusible_slice_test

Copybara import of the project:

--
0d307384bff386d5182f89ae5a5422f8ca1a1290 by Dragan Mladjenovic <[email protected]>:

[ROCm] Fix //xla/tests:complex_unary_op_test and //xla/service/gpu/tests:gpu_input_fusible_slice_test

Merging this change closes #19484

PiperOrigin-RevId: 698374588

… CUDA_ERROR_ILLEGAL_ADDRESS in (micro)benchmarks with FP8 Triton kernels during exhaustive autotuning.

PiperOrigin-RevId: 698387396

PiperOrigin-RevId: 698388778

PiperOrigin-RevId: 698388866

PiperOrigin-RevId: 698389323

What this change does is it:

 1. Identifies all `kTfLiteBuiltinDequantize` nodes converting `kTfLiteFloat16` to `kTfLiteFloat32` and plugging into a `kTfLiteBuiltinFullyConnected`, `kTfLiteBuiltinConv2d`, or `kTfLiteBuiltinDepthwiseConv2d` node.
 2. Re-maps XNNPACK tensors pointing to the `kTfLiteFloat32` output to point to the original `kTfLiteFloat16` input.

The `kTfLiteFloat16` weights/filters and biases are handled by XNNPACK directly.

PiperOrigin-RevId: 698395748

…g_util by using `PropagateShardingAlongDimsAndReplicateOthers`. This is a no-op change.

PiperOrigin-RevId: 698403022

…ernal to a MutableOpResolver.

PiperOrigin-RevId: 698407108

PiperOrigin-RevId: 698410181

We need to adjust the heuristic because before our emitter had an issue that prevented Triton from doing proper layout optimizations. It was fixed in openxla/xla@7280b9a.

We needed to use higher number of warps (up to 32) before to cover the lack of layout optimization, but now it can cause performance regressions, because Triton likes to insert shmem usage and barrier syncs.

PiperOrigin-RevId: 698416298

…ather/scatter instructions. Implicit batching dims are also known as index parallel dims.

Update `GetGatherScatterBatchParallelDims` accordingly. The sharding propagation and spmd partitioner will process explicit and implicit batching dims separately.

PiperOrigin-RevId: 698421986

Minimal XLA:CPU runtime implementation optimized for low latency inference.

--------------------------------------------------------------
Benchmark                    Time             CPU   Iterations
--------------------------------------------------------------
BM_NanoRtAddScalars       84.8 ns         84.8 ns      8277118
BM_NanoRtFibonacci        81.1 ns         81.1 ns      8468298
BM_PjRtAddScalars         1517 ns         1517 ns       460076
BM_PjRtFibonacci          1523 ns         1523 ns       460415

PiperOrigin-RevId: 698426607

PiperOrigin-RevId: 698427377

name                     old cpu/op   new cpu/op   delta
BM_CountDownSuccess/4    97.6ns ± 2%  97.9ns ± 1%    ~     (p=0.841 n=5+5)
BM_CountDownSuccess/8     123ns ± 2%   122ns ± 1%    ~     (p=0.548 n=5+5)
BM_CountDownSuccess/16    171ns ± 1%   172ns ± 2%    ~     (p=0.548 n=5+5)
BM_CountDownSuccess/32    270ns ± 1%   271ns ± 1%    ~     (p=0.310 n=5+5)
BM_CountDownError/4       215ns ± 1%   212ns ± 3%    ~     (p=0.310 n=5+5)
BM_CountDownError/8       309ns ± 2%   307ns ± 1%    ~     (p=0.421 n=5+5)
BM_CountDownError/16      500ns ± 1%   496ns ± 2%    ~     (p=0.421 n=5+5)
BM_CountDownError/32      888ns ± 1%   885ns ± 2%    ~     (p=0.548 n=5+5)

PiperOrigin-RevId: 698431683

…ed) in CUDA BUILD file.

PiperOrigin-RevId: 698444212

…esolved)

PiperOrigin-RevId: 698444339

	Todo(resolved)

PiperOrigin-RevId: 698445171

	Todo(resolved)

PiperOrigin-RevId: 698445177

PiperOrigin-RevId: 698448973

PiperOrigin-RevId: 698452075

Currently the transposed convolution is orders of magnitude slower than the regular one. Ideally performance should be similar. Detailed results:

----------------------------------------------------------------------------------
Benchmark                                        Time             CPU   Iterations
----------------------------------------------------------------------------------
BM_Conv2DStrided/process_time              3737222 ns     41608631 ns           16
BM_Conv2DTransposedStrided/process_time  590079914 ns   1.0847e+10 ns            1

PiperOrigin-RevId: 698453016

The correct output dimension when dumped to HLO text is `bf0`, where `f` means the output feature dimension. There is no dimension called `o`.

PiperOrigin-RevId: 698453240

… State

By keeping AsyncValueRef as a part of the State we avoid one extra reference counting operation when copying CountDownAsyncValue (and we expect to copy it `cnt` times).

name                     old cpu/op   new cpu/op   delta
BM_CountDownSuccess/8    95.8ns ± 4%  81.7ns ± 1%  -14.64%  (p=0.000 n=40+35)
BM_CountDownSuccess/16    142ns ± 1%   127ns ± 1%  -10.05%  (p=0.000 n=37+38)
BM_CountDownSuccess/32    229ns ± 2%   216ns ± 1%   -5.56%  (p=0.000 n=40+38)
BM_CountDownError/4       165ns ± 1%   152ns ± 2%   -7.65%  (p=0.000 n=39+40)
BM_CountDownError/8       238ns ± 2%   225ns ± 1%   -5.65%  (p=0.000 n=40+38)
BM_CountDownError/16      388ns ± 2%   369ns ± 2%   -4.77%  (p=0.000 n=40+36)
BM_CountDownError/32      684ns ± 1%   666ns ± 1%   -2.50%  (p=0.000 n=38+38)

PiperOrigin-RevId: 698454410

…en calls to `Delegate::PrepareOpsToDelegate`.

PiperOrigin-RevId: 698455074

PiperOrigin-RevId: 698462670

PiperOrigin-RevId: 698466696

… compiler.

Imported from GitHub PR openxla/xla#19237

Copybara import of the project:

--
177f911fd4c6af86c25aba2e38ea09767477be03 by Ilia Sergachev <[email protected]>:

[GPU] Fix passing of key-value store handle from client to compiler.

--
ec2b96ccdf8cd81abdc25f3cff2bdf65df455219 by Ilia Sergachev <[email protected]>:

use allowed_devices instead of CUDA_VISIBLE_DEVICES

--
77ba9fd7b172052269fafd1a1970d58d1d803a59 by Ilia Sergachev <[email protected]>:

skip the added test on pre-Ampere GPUs

Merging this change closes #19237

PiperOrigin-RevId: 698469112

PiperOrigin-RevId: 698473752

PiperOrigin-RevId: 698474274

No functional change is intended but it generates less IR.

PiperOrigin-RevId: 698477060

…er_interval_comparator into msa/utils.

PiperOrigin-RevId: 698481492

PiperOrigin-RevId: 698485833

As of this CL, all array operations (except `IsDeleted()`) are asynchronous.

This CL also makes the following drive-by changes:

1. Version management is getting refactored to use an enum and a header file
within /common.

2. All error responses from the server (except connection
terminations, which follow the previous behavior) are now printed out as a
WARNING.

PiperOrigin-RevId: 698491308

PiperOrigin-RevId: 698494677

PiperOrigin-RevId: 698502341

Also add a bit more comments and re-organize some things.

PiperOrigin-RevId: 698507311

…e the .td definition.

PiperOrigin-RevId: 698510501

PiperOrigin-RevId: 698519430

…ion poitners. check for correct union types in tensor cc api

PiperOrigin-RevId: 698529069