Disabling xnnpack except for dynamic quantlized linear (#391)

Summary:
In executorch this path was slower compared to using optimized ops. We
did not debug that further, but for now just disabled it.

Test Plan:
CI

Reviewers:

Subscribers:

Tasks:

Tags:

export_et.py

@@ -97,7 +97,6 @@ def export_model(model, device, output_path, args=None) -> str:  # noqa: C901
             edge_compile_config=edge_config,
         )
     edge_manager = edge_manager.to_backend(XnnpackDynamicallyQuantizedPartitioner())
-    edge_manager = edge_manager.to_backend(XnnpackPartitioner())
     export_program = edge_manager.to_executorch(
         ExecutorchBackendConfig(
             extract_constant_segment=True,