Server formatting tests are flaky on the main channel

[johnpryan]

See this comment for details: #3091 (comment)

[DanTup]

Hmm, that's concerning. An example failure is here:

https://github.com/dart-lang/dart-pad/actions/runs/11861582966/job/33059207802

❌ test/server_test.dart: server format no changes (failed)
  Expected: 'void main() {\n'
              '  print(\'hello world\');\n'
              '}\n'
              ''
    Actual: 'void main() {\n'
              '  print(\'hello world\');\n'
              '}\n'
              '}\n'
              ''
     Which: is different. Both strings start the same, but the actual value also has the following trailing characters: }\n
  
  package:matcher              expect
  test/server_test.dart 172:7  defineTests.<fn>.<fn>

If the issue is in the analysis server, I can try to repro and take a look at this next week.

[DanTup]

I couldn't reproduce this locally - @devoncarew does it fail for you locally? I made a PR that spits out the traffic to see exactly what the server is returning:

#3094

Is it possible to run on the bots and/or locally to see what the server traffic looks like?

[DanTup]

This does look like a server bug... The previous test sets the content of the file to 38 characters and the log shows that analysis starts (but it doesn't seem to complete).

The next test sets the content to 40 characters (it adds two additional newlines) and the format result is to replace 38 characters (not 40). There was no change in analysis status during this period.

// "server format" test sets single-line content (38 chars) and formats
2024-11-18T14:28:45.0773793Z ==> {"id":"21","method":"analysis.updateContent","params":{"files":{"/home/runner/work/dart-pad/dart-pad/pkgs/dart_services/project_templates/flutter_project/main.dart":{"type":"add","content":"void main() { print('hello world'); }\n"}}}}
2024-11-18T14:28:45.0806324Z <== {"id":"21","result":{}}
2024-11-18T14:28:45.0807657Z ==> {"id":"22","method":"analysis.setPriorityFiles","params":{"files":["/home/runner/work/dart-pad/dart-pad/pkgs/dart_services/project_templates/flutter_project/main.dart"]}}
2024-11-18T14:28:45.0808648Z <== {"event":"server.status","params":{"analysis":{"isAnalyzing":true}}}
2024-11-18T14:28:45.0843845Z <== {"event":"analysis.errors","params":{"file":"/home/runner/work/dart-pad/dart-pad/pkgs/dart_services/project_templates/flutter_project/main.dart","errors":[]}}
2024-11-18T14:28:45.0875197Z <== {"id":"22"}
2024-11-18T14:28:45.0884778Z ==> {"id":"23","method":"edit.format","params":{"file":"/home/runner/work/dart-pad/dart-pad/pkgs/dart_services/project_templates/flutter_project/main.dart","selectionOffset":0,"selectionLength":0}}
2024-11-18T14:28:45.1182301Z <== {"id":"23","result":{"edits":[{"offset":0,"length":38,"replacement":"void main() {\n  print('hello world');\n}\n"}],"selectionOffset":0,"selectionLength":0}}
2024-11-18T14:28:45.1240449Z ✅ test/server_test.dart: server format


// "server format no changes" test sets multi-line content (40 chars) and formats
2024-11-18T14:28:45.1266207Z ==> {"id":"24","method":"analysis.updateContent","params":{"files":{"/home/runner/work/dart-pad/dart-pad/pkgs/dart_services/project_templates/flutter_project/main.dart":{"type":"add","content":"void main() {\n  print('hello world');\n}\n"}}}}
2024-11-18T14:28:45.1293815Z <== {"id":"24","result":{}}
2024-11-18T14:28:45.1297918Z ==> {"id":"25","method":"analysis.setPriorityFiles","params":{"files":["/home/runner/work/dart-pad/dart-pad/pkgs/dart_services/project_templates/flutter_project/main.dart"]}}
2024-11-18T14:28:45.1323099Z <== {"id":"25"}
2024-11-18T14:28:45.1325528Z ==> {"id":"26","method":"edit.format","params":{"file":"/home/runner/work/dart-pad/dart-pad/pkgs/dart_services/project_templates/flutter_project/main.dart","selectionOffset":0,"selectionLength":0}}

// Format results has a bogus length of 38, but we expect 40. The results in leaving the last two characters of content that should be replaced.
2024-11-18T14:28:45.1360243Z <== {"id":"26","result":{"edits":[{"offset":0,"length":38,"replacement":"void main() {\n  print('hello world');\n}\n"}],"selectionOffset":0,"selectionLength":0}}



2024-11-18T14:28:45.1773383Z ##[group]❌ test/server_test.dart: server format no changes (failed)
2024-11-18T14:28:45.1774310Z Expected: 'void main() {\n'
2024-11-18T14:28:45.1774886Z             '  print(\'hello world\');\n'
2024-11-18T14:28:45.1775470Z             '}\n'
2024-11-18T14:28:45.1775909Z             ''
2024-11-18T14:28:45.1776436Z   Actual: 'void main() {\n'
2024-11-18T14:28:45.1776943Z             '  print(\'hello world\');\n'
2024-11-18T14:28:45.1777421Z             '}\n'
2024-11-18T14:28:45.1777803Z             '}\n'
2024-11-18T14:28:45.1778202Z             ''
2024-11-18T14:28:45.1779230Z    Which: is different. Both strings start the same, but the actual value also has the following trailing characters: }\n
2024-11-18T14:28:45.1780253Z 

It's not clear why this happens on the bots but not for my locally (also on latest Flutter master).

[DanTup]

I reproduced this in a server test - I've filed dart-lang/sdk#57120

[DanTup]

This is fixed by dart-lang/sdk@be7a40d

[johnpryan]

Thanks for digging into this!