Skip to content

Commit

Permalink
rename to parse_with_layout
Browse files Browse the repository at this point in the history
  • Loading branch information
@Sdddell
Sdddell committed Oct 30, 2024
1 parent 439fdc3 commit 766b7cf
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 17 deletions.
4 changes: 2 additions & 2 deletions any_parser/any_parser.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ class ProcessType(Enum):
TABLE = "table"
FILE_REFINED = "file_refined"
FILE_REFINED_QUICK = "file_refined_quick"
FILE_ADVANCED = "file_advanced"
PARSE_WITH_LAYOUT = "parse_with_layout"


class AnyParser:
Expand Down Expand Up @@ -223,7 +223,7 @@ def async_extract(
elif model == ModelType.PRO:
process_type = ProcessType.FILE_REFINED_QUICK
elif model == ModelType.ADVANCED:
process_type = ProcessType.FILE_ADVANCED
process_type = ProcessType.PARSE_WITH_LAYOUT
else:
return "Error: Invalid model type", None

Expand Down
29 changes: 14 additions & 15 deletions examples/async_advanced_full_context.ipynb → examples/async_parse_with_layout.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -23,16 +23,16 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"ap = AnyParser(api_key=\"...\")"
"ap = AnyParser(api_key=\"S4iyw7RAEE8CTGkVgHYeI8nsTmSALI1U2HXvAN6j\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -42,13 +42,14 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Waiting for response...\n",
"Waiting for response...\n"
]
}
Expand All @@ -59,7 +60,7 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 6,
"metadata": {},
"outputs": [
{
Expand All @@ -71,12 +72,12 @@
"<tr><td> </td><td>latency </td><td>(ms) </td></tr>\n",
"<tr><td>participants</td><td>mean </td><td>99th percentile</td></tr>\n",
"<tr><td>1 </td><td>17.0 +1.4 </td><td>75.0 34.9 </td></tr>\n",
"<tr><td>2 </td><td>24.5 +2.5 </td><td>87.6 35.9 </td></tr>\n",
"<tr><td>2 </td><td>24.5 +2.5 </td><td>87.6 +35.9 </td></tr>\n",
"<tr><td>5 </td><td>31.5 +6.2 </td><td>104.5 52.2 </td></tr>\n",
"<tr><td>10 </td><td>30.0 +3.7 </td><td>95.6 25.4 </td></tr>\n",
"<tr><td>10 </td><td>30.0 +3.7 </td><td>95.6 +25.4 </td></tr>\n",
"<tr><td>25 </td><td>35.5 +5.6 </td><td>100.4 42.7 </td></tr>\n",
"<tr><td>50 </td><td>42.7 4.1 </td><td>93.7 22.9 </td></tr>\n",
"<tr><td>100 </td><td>71.4 +7.6 </td><td>131.2 11.6 </td></tr>\n",
"<tr><td>100 </td><td>71.4 7.6 </td><td>131.2 +17.6 </td></tr>\n",
"<tr><td>200 </td><td>150.5 +11.0</td><td>320.3 35.1 </td></tr>\n",
"</tbody>\n",
"</table>\n",
Expand All @@ -91,19 +92,17 @@
"\n",
"5.2 Availability\n",
"\n",
"Figure 5 illustrates the availability benefits of running Spanner in multiple datacenters. It shows the results of three experiments on throughput in the presence of datacenter failure, all of which are overlaid onto the same time scale. The test universe consisted of 5 zones \\(Z_i\\), each of which had 25 spanservers. The test database was sharded into 1250 Paxos groups, and 100 test clients constantly issued non-snapshot reads at an aggregate rate of 50K reads/second. All of the leaders were explicitly placed in \\(Z_1\\). Five seconds into each test, all of the servers in one zone were killed: non-leader kills \\(Z_2\\); leader-hard kills \\(Z_1\\); leader-soft kills \\(Z_1\\), but it gives notifications to all of the servers that they should handoff leadership first.\n",
"Figure 5 illustrates the availability benefits of running Spanner in multiple datacenters. It shows the results of three experiments on throughput in the presence of datacenter failure, all of which are overlaid onto the same time scale. The test universe consisted of 5 zones Zi, each of which had 25 spanservers. The test database was sharded into 1250 Paxos groups, and 100 test clients constantly issued non-snapshot reads at an aggregate rate of 50K reads/second. All of the leaders were explicitly placed in Z1. Five seconds into each test, all of the servers in one zone were killed: non-leader kills Z2; leader-hard kills Z1; leader-soft kills Z1, but it gives notifications to all of the servers that they should handoff leadership first.\n",
"\n",
"Killing \\(Z_2\\) has no effect on read throughput. Killing \\(Z_1\\) while giving the leaders time to handoff leadership to a different zone has a minor effect: the throughput drop is not visible in the graph, but is around 3-4%. On the other hand, killing \\(Z_1\\) with no warning has a severe effect: the rate of completion drops almost to 0. As leaders get re-elected, though, the throughput of the system rises to approximately 100K reads/second because of two artifacts of our experiment: there is extra capacity in the system, and operations are queued while the leader is unavailable. As a result, the throughput of the system rises before leveling off again at its steady-state rate.\n",
"Killing Z2 has no effect on read throughput. Killing Z1 while giving the leaders time to handoff leadership to a different zone has a minor effect: the throughput drop is not visible in the graph, but is around 3-4%. On the other hand, killing Z1 with no warning has a severe effect: the rate of completion drops almost to 0. As leaders get re-elected, though, the throughput of the system rises to approximately 100K reads/second because of two artifacts of our experiment: there is extra capacity in the system, and operations are queued while the leader is unavailable. As a result, the throughput of the system rises before leveling off again at its steady-state rate.\n",
"\n",
"We can also see the effect of the fact that Paxos leader leases are set to 10 seconds. When we kill the zone, the leader-lease expiration times for the groups should be evenly distributed over the next 10 seconds. Soon after each lease from a dead leader expires, a new leader is elected. Approximately 10 seconds after the kill time, all of the groups have leaders and throughput has recovered. Shorter lease times would reduce the effect of server deaths on availability, but would require greater amounts of lease-renewal network traffic. We are in the process of designing and implementing a mechanism that will cause slaves to release Paxos leader leases upon leader failure.\n",
"\n",
"5.3 TrueTime\n",
"\n",
"Two questions must be answered with respect to TrueTime: is ε truly a bound on clock uncertainty, and how bad does ε get? For the former, the most serious problem would be if a local clock’s drift were greater than 200μsec/sec: that would break assumptions made by TrueTime. Our machine statistics show that bad CPUs are 6 times more likely than bad clocks. That is, clock issues are extremely infrequent, relative to much more serious hardware problems. As a result, we believe that TrueTime’s implementation is as trustworthy as any other piece of software upon which Spanner depends.\n",
"\n",
"Figure 6 presents TrueTime data taken at several thousand spanservers across three datacenters up to 2200\n",
"Two questions must be answered with respect to TrueTime: is ε truly a bound on clock uncertainty, and how bad does ε get? For the former, the most serious problem would be if a local clock’s drift were greater than 200us/sec: that would break assumptions made by TrueTime. Our machine statistics show that bad CPUs are 6 times more likely than bad clocks. That is, clock issues are extremely infrequent, relative to much more serious hardware problems. As a result, we believe that TrueTime’s implementation is as trustworthy as any other piece of software upon which Spanner depends.\n",
"\n",
"![<@mask_p0_e1_figure(timeout=1h)>](https://anyparser-realtime-test-j-assetsconstructfilebucke-2wg0ln280yvz.s3.amazonaws.com/result_file_advanced/async_S4iyw7RAEE8CTGkVgHYeI8nsTmSALI1U2HXvAN6j/2024/10/29/test_1figure_1table_ca33eeda-23ac-4a1b-a6e3-17317cc54481.png/%3C%40mask_p0_e1_figure_s3%3E.png?AWSAccessKeyId=ASIAXM24X76XPGTHDL75&Signature=1F2FBNLAat4%2FS9v2sCgufTHbnhA%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEN7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCICyyNIpApgz38DKh44HxjxiimKaZtsThrK4LHWp2pvFNAiEAqCNitanW6d9lUNPAFqHPcNsPu9U8fJb2b3tPY5Doagoq0wMIVxAAGgw1MDg2MTEyNjQ0MzAiDG5B0BM7FjSAapU5ziqwA4IRQu%2FLS9Z8jmgEdcO70c5kHJCZ0f5dx3q0pY8Ahxcqnu9KB0omgTjIf7kaGDVI%2FwccE%2B97joOhWo2zERnRM5gWIJAnWvYg5iXFBs3YXbF6%2BaSl8sBQgxRxSIhJZltSk2S2zaKlkLcrfMy17%2BX0rqp1YqZDMBk4dPoaZpTNh1oqwSDEhcKPW7F04ZRUTZcsAYQHE%2BAu7doeaGXCeSI3oViHYOU%2BOdpAjBgWCpr4beZYlwVdUvYU%2F6BqNcauxidyBFMPRQV9ZloBdhkcmtvHqFRr%2Fl3z9s7t4JdXXnIRSxssKlBrlsL8J%2FCCfCxzySA24%2Fp%2BNkiefHcMRA12rORqaV%2BawZXGHeTIjpeLAuARcyDBoe2C7g36Qn%2FRNRAJeM3lJUoJpVhJt6PvaZzL8%2B%2Bu0hV%2FGOwDwh1hL5Ltpr6DE7FEVCem%2Bmmg066QG01fM1XKYCSaCINbRspifY4jDgZvLcFpcFoXVhdKisEehpys6ESUocjzbip9SqdQAaIbRifyCueDauHd5vC%2FgO0kZgTYfNVCqi%2FvQI0iOtpdWEaK%2F3h9XrRkvfKkkEuGIhyfW4NljjCw64G5BjqeAQcgJHPslWjoHbul7HHc6ULF0OkI38W4u%2F4GxmNepvntBsSy3zcSn43vGzbxzczK37VgDvK%2BTNy4t3tyQfUZYI%2Fw308Y1CVFRVyt7wTDe5jHzDUVLtx8jO0eP4kj8CvjWTAHPRSbLiSMmKF0noEbPt16i6pBjcNQltSqApB%2FbySYQxndf09KyPLD8xS%2Fd3k5TM5B5InfGE5KyVnSZEor&Expires=1730184369)\n",
"![<@mask_p0_e1_figure(timeout=1h)>](https://anyparser-realtime-test-j-assetsconstructfilebucke-2wg0ln280yvz.s3.amazonaws.com/result_parse_with_layout/async_S4iyw7RAEE8CTGkVgHYeI8nsTmSALI1U2HXvAN6j/2024/10/30/test_1figure_1table_215dd6ed-92a2-4636-8dc0-5636c689bf4b.png/%3C%40mask_p0_e1_figure_s3%3E.png?AWSAccessKeyId=ASIAXM24X76XBW2GFBFU&Signature=FyIrRoyyRfiKirQbeuVzNzXAowQ%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEPT%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQCRUsV%2FHrEppoMWhVoou%2Ft2j%2FrgGV61zgYoeuLwFbKLcAIgKfL4H8nBkzZt2QWR0vm8ZjbKGJclV64yFzh%2FhHChK7Mq0wMIbRAAGgw1MDg2MTEyNjQ0MzAiDDPz9lhEp9Z9OwEe6SqwA6bt%2BuJ8wJD2UQ2OJpUYVwzjhPPKupLRyj0QWt6zNTg%2F%2BKYcLg6mx7s47rmNFpVAjr28CFcX8DxU9DvgILucIPPs2FCvxMoGpXGmrerdRONK1hbSWiEsaiVd7%2B4AW27R5omTB%2F1%2Fk84WUBZRuWPCGn8dZPqAZaBM4dSyUMBjnoTKBmaytwbxTOLQ9mvil%2Fj7JjZaeOtuBWGN8M8GPikXwTtXiJDFqofsWn8hRzMsMYRrueamEn%2BRZ07l9FIie4VRYNYuQDE5DkVI5misl9lpXY1OawATogpCj8SBI%2BwqZUpVe89qj67cMCKH78vAdBqbteNy14M36DIhdiRrVOXzoyibQW4I4jp9wtDQdSbPjS9RqRC36CLfseN%2BVzjWzYxhGoI2sPF5NMPHVGPMSq6VcCA3G%2FW85FLER6XRKCdM03%2FLrt1G7ocowIsbVHE%2BP%2F0jh5Bg8ZQyd%2FUb0yS%2FJQMbQzM72qZOP8nMfl4eytljo4vMDwwP9485uPqM1wcHGrMtk48vqlgcM9Br5cQtikV3q4Q8O3ND9G5xmICPO4LhfE%2B9XeKkXdekRBQ7qTj6ub%2BxEDDS2oa5BjqeAXJbS7XvEc73%2BsWZwamhZLO5Nia3uJDJY5qGjDGEieB%2FhtpLyszAJGHLkJKSaGBM8kwyiPGybwidi7vFcGBZ08CBk7RXVQFpFWmSKtRfbVqOCubcsmgfTUjb7UOfRz3swIvEt3kjazXLowhvy9U1%2B3ISQohEe80zKSJHbc0nE75%2F%2BP97Cl6%2FAVsFqKMZPLcsDrooFR3M5WBPUqebcJQ%2F&Expires=1730264238)\n",
"\n",
"Figure 5: Effect of killing servers on throughput.\n"
],
Expand Down

0 comments on commit 766b7cf

Please sign in to comment.