-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add advanced full context and example
- Loading branch information
Showing
4 changed files
with
149 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Install the libraries (ipython is used for displaying markdown in this demo)\n", | ||
| "# !pip3 install --upgrade ipython\n", | ||
| "# !pip3 install --upgrade any-parser" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 2, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from IPython.display import display, Markdown\n", | ||
| "from any_parser import AnyParser, ModelType" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 3, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "ap = AnyParser(api_key=\"...\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 11, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "file_path = \"./sample_data/test_1figure_1table.png\"\n", | ||
| "file_id = ap.async_extract(file_path, ModelType.ADVANCED, {})" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 12, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Waiting for response...\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "markdown_output = ap.async_fetch(file_id=file_id)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 13, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "data": { | ||
| "text/markdown": [ | ||
| "\n", | ||
| "<table>\n", | ||
| "<tbody>\n", | ||
| "<tr><td> </td><td>latency </td><td>(ms) </td></tr>\n", | ||
| "<tr><td>participants</td><td>mean </td><td>99th percentile</td></tr>\n", | ||
| "<tr><td>1 </td><td>17.0 +1.4 </td><td>75.0 34.9 </td></tr>\n", | ||
| "<tr><td>2 </td><td>24.5 +2.5 </td><td>87.6 35.9 </td></tr>\n", | ||
| "<tr><td>5 </td><td>31.5 +6.2 </td><td>104.5 52.2 </td></tr>\n", | ||
| "<tr><td>10 </td><td>30.0 +3.7 </td><td>95.6 25.4 </td></tr>\n", | ||
| "<tr><td>25 </td><td>35.5 +5.6 </td><td>100.4 42.7 </td></tr>\n", | ||
| "<tr><td>50 </td><td>42.7 4.1 </td><td>93.7 22.9 </td></tr>\n", | ||
| "<tr><td>100 </td><td>71.4 +7.6 </td><td>131.2 11.6 </td></tr>\n", | ||
| "<tr><td>200 </td><td>150.5 +11.0</td><td>320.3 35.1 </td></tr>\n", | ||
| "</tbody>\n", | ||
| "</table>\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "Table 4: Two-phase commit scalability. Mean and standard deviations over 10 runs.\n", | ||
| "\n", | ||
| "CPUs. Snapshot reads can execute at any up-to-date replicas, so their throughput increases almost linearly with the number of replicas. Single-read read-only transactions only execute at leaders because timestamp assignment must happen at leaders. Read-only-transaction throughput increases with the number of replicas because the number of effective spanservers increases: in the experimental setup, the number of spanservers equaled the number of replicas, and leaders were randomly distributed among the zones. Write throughput benefits from the same experimental artifact (which explains the increase in throughput from 3 to 5 replicas), but that benefit is outweighed by the linear increase in the amount of work performed per write, as the number of replicas increases.\n", | ||
| "\n", | ||
| "Table 4 demonstrates that two-phase commit can scale to a reasonable number of participants: it summarizes a set of experiments run across 3 zones, each with 25 spanservers. Scaling up to 50 participants is reasonable in both mean and 99th-percentile, and latencies start to rise noticeably at 100 participants.\n", | ||
| "\n", | ||
| "5.2 Availability\n", | ||
| "\n", | ||
| "Figure 5 illustrates the availability benefits of running Spanner in multiple datacenters. It shows the results of three experiments on throughput in the presence of datacenter failure, all of which are overlaid onto the same time scale. The test universe consisted of 5 zones \\(Z_i\\), each of which had 25 spanservers. The test database was sharded into 1250 Paxos groups, and 100 test clients constantly issued non-snapshot reads at an aggregate rate of 50K reads/second. All of the leaders were explicitly placed in \\(Z_1\\). Five seconds into each test, all of the servers in one zone were killed: non-leader kills \\(Z_2\\); leader-hard kills \\(Z_1\\); leader-soft kills \\(Z_1\\), but it gives notifications to all of the servers that they should handoff leadership first.\n", | ||
| "\n", | ||
| "Killing \\(Z_2\\) has no effect on read throughput. Killing \\(Z_1\\) while giving the leaders time to handoff leadership to a different zone has a minor effect: the throughput drop is not visible in the graph, but is around 3-4%. On the other hand, killing \\(Z_1\\) with no warning has a severe effect: the rate of completion drops almost to 0. As leaders get re-elected, though, the throughput of the system rises to approximately 100K reads/second because of two artifacts of our experiment: there is extra capacity in the system, and operations are queued while the leader is unavailable. As a result, the throughput of the system rises before leveling off again at its steady-state rate.\n", | ||
| "\n", | ||
| "We can also see the effect of the fact that Paxos leader leases are set to 10 seconds. When we kill the zone, the leader-lease expiration times for the groups should be evenly distributed over the next 10 seconds. Soon after each lease from a dead leader expires, a new leader is elected. Approximately 10 seconds after the kill time, all of the groups have leaders and throughput has recovered. Shorter lease times would reduce the effect of server deaths on availability, but would require greater amounts of lease-renewal network traffic. We are in the process of designing and implementing a mechanism that will cause slaves to release Paxos leader leases upon leader failure.\n", | ||
| "\n", | ||
| "5.3 TrueTime\n", | ||
| "\n", | ||
| "Two questions must be answered with respect to TrueTime: is ε truly a bound on clock uncertainty, and how bad does ε get? For the former, the most serious problem would be if a local clock’s drift were greater than 200μsec/sec: that would break assumptions made by TrueTime. Our machine statistics show that bad CPUs are 6 times more likely than bad clocks. That is, clock issues are extremely infrequent, relative to much more serious hardware problems. As a result, we believe that TrueTime’s implementation is as trustworthy as any other piece of software upon which Spanner depends.\n", | ||
| "\n", | ||
| "Figure 6 presents TrueTime data taken at several thousand spanservers across three datacenters up to 2200\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "Figure 5: Effect of killing servers on throughput.\n" | ||
| ], | ||
| "text/plain": [ | ||
| "<IPython.core.display.Markdown object>" | ||
| ] | ||
| }, | ||
| "metadata": {}, | ||
| "output_type": "display_data" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "display(Markdown(markdown_output))" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "any", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.10" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.