-
Notifications
You must be signed in to change notification settings - Fork 78
Data captured
aliases categories, claims, contributors, description, exhtml, exrest, extext, extract, files, html, image, infobox label, languages, links, lead, length, links, modified, pageid, parsetree, properties, random, title, url, url_raw, views, watchers, what, wikibase, wikidata, wikidata_url, wikitext
<list> alternative names for items in Wikidata
This is the list of aliases from Wikidata.
Example:
>>> gandhi.data['aliases'][:3]
[u'M K Gandhi',
u'Mohandas Gandhi',
u'Bapu']Methods: get_wikidata()
<list> all categories page belongs to
This is a list of all categories that appear on the page and which the page belongs to. We get this from the Mediawiki API:Categories module.
Example:
>>> gandhi.data['categories'][:3]
[u'Category:1869 births',
u'Category:1948 deaths',
u'Category:19th-century Indian lawyers']
Methods: get_query()
<dict> Q entity claims that resolve to wikidata labels
Claims are Q entities that have labels in different languages. For example Q1 is "universe" in English. Claims are collected when fetching wikidata. They are later resolved into labels for the specified language to populate the wikidata page attribute.
Example:
>>> gandhi.claims {u'Q129286': 'citizenship', u'Q5': 'instance', u'Q6512732': 'category', u'Q668': 'citizenship'}
Methods: get_claims()
<int> total number of contributors to this page
This is the total of logged-in AND anonymous contributors from Mediawiki API:Contributors. Fascinating!
Example:
gandhi.contributors: 2118
Methods: get_query()
<str> short description
This is a short description of the page from Mediawiki or Wikidata. When it is available it is often very enlightening.
Example:
ghandi.description: pre-eminent leader of Indian nationalism during British-ruled India
Methods: get_query(), get_wikidata()
<str> RESTBase page extract in HTML
This is the RESTBase "extract_html" (summary) in limited HTML. It does not contain interwiki links, citations or infoboxes. It is basically a trunctated version of page.data['extract'].
Example:
gandhi.exhtml: <str(1036)> <p><b>Mohandas Karamchand Gandhi</b> (<span></...
Methods: get_restbase('/page/summary')
<str> RESTBase page extract in plain text
This is the RESTBase "extract" (summary) in plain text. It is basically a truncated version of page.data['extext'].
Example:
gandhi.exrest: <str(889)> Mohandas Karamchand Gandhi (; Hindustani: [ˈmoː...
Methods: get_restbase('/page/summary')
<str> page extract in plain text
This is the lead section, or summary, of the page in plain text. It does not include infoboxes, and some data is removed by the API.
Example:
gandhi.extext: <str(2987)> **Mohandas Karamchand Gandhi** (; Hindustani: ...
Methods: get_query()
<str> page extract in limited HTML
This is the lead section, or summary, of the page in limited HTML. It is simple markup only; no wikilinks, citations, infoboxes, etc.
Example:
gandhi.extract: <str(3192)> <p><b>Mohandas Karamchand Gandhi</b> (<span><...
Methods: get_query()
<list> list of files embedded in this page
This is the list of embedded (image, audio, video) files included on this page from Mediawiki API:Images. Awesome!
Example:
gandhi.files: <list(53)>
>>> [x for x in p.files if not x.endswith('jpg') and not x.endswith('svg')]
[u'File:Mohandas Karamchand Gandhi pronunciation 3.oga',
u'File:Salt March.ogg',
u'File:Salt March.ogv',
u'File:Socrates.png',
u'File:Young India.png']Methods: get_query()
<str> page content in full HTML
This is the most performant way to get page HTML outside of running your own Mediawiki instance. It is verbatim what Mediawiki is serving for that page.
Example:
> gandhi.html: <str(1394338)> <!DOCTYPE html><html prefix="dc: http://purl....
Methods: get_restbase('/page/html')
<list> representative image(s) for this page
The epitome ("single most appropriate") image data contained in each API response is stored in this attribute with a kind label. They are often all the same image file. These are NOT all the images/files contained in a page—that would be page.data['files']—, only the so-called PageImage that aims to be a representative image for the page. See the Images documentation for more details.
Example:
>>> gandhi.pageimage()
['query-pageimage',
'query-thumbnail',
'parse-image',
'wikidata-image',
'rest-image',
'rest-thumb']
>>> gandhi.pageimage('thumb')
{'file': u'Portrait_Gandhi.jpg',
u'height': 240,
'kind': 'query-thumbnail',
'url': u'https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Portrait_Gandhi.jpg/160px-Portrait_Gandhi.jpg',
u'width': 160}Methods: get_imageinfo(), get_parse(), get_query(), get_restbase(), get_wikidata()
<dict> parsed infobox data
This attribute contains Infobox template data extracted from a page's parsetree. Unfortunately, there is usually more data available from a page's infobox than from wikidata. See the Infoboxes documentation for details.
Example:
>>> gandhi.infobox {'alma_mater': [[University College London]]<ext><name>ref</name><attr/><i... 'alt': u'The face of Gandhi in old age\u2014smiling, wearing glasses, and ... 'birth_date': '{{Birth date|df|=|yes|1869|10|2}}', 'birth_name': 'Mohandas Karamchand Gandhi', 'birth_place': [[Porbandar State]], [[Kathiawar Agency]], [[British India... 'children': '{{hlist|[[Harilal Gandhi|Harilal]]|[[Manilal Gandhi|Manilal]]... 'death_cause': '[[Assassination of Mahatma Gandhi|Assassination]]', 'death_date': '{{Death date and age|df|=|yes|1948|1|30|1869|10|2}}', 'death_place': 'New Delhi, [[Delhi]], [Dominion of India] (now India)', 'father': '[[Karamchand Uttamchand Gandhi|Karamchand Gandhi]]', 'honorific_prefix': '[[Mahatma]]', 'image': 'MKGandhi.jpg', 'known_for': [[Indian Independence Movement]],<br>[[Peace movement]]', 'mother': 'Putlibai Gandhi', 'movement': '[[Indian independence movement]]', 'name': 'Mohandas Karamchand Gandhi', 'occupation': '{{hlist|Lawyer|Politician|Activist|Writer|Soldier}}', 'other_names': 'Mahatma Gandhi, Bapu, Gandhiji', 'party': '[[Indian National Congress]]', 'resting_place': [[Raj Ghat and associated memorials|Raj Ghat]], Delhi', 'signature': 'Mohandas K. Gandhi signature.svg', 'spouse': '{{marriage|[[Kasturba Gandhi]]|1883|1944|end|=|died}}'}
Methods: get_parse()
<str> Wikidata label
This is the Wikidata label (common name) in the language specified.
Example:
gandhi.label: Mahatma Gandhi
Methods: get_query(), get_wikidata()
<list> languages available
This is the list of languages that this page can be found in on other Wikipedias from Mediawiki API:Langlinks. Each entry contains the language code and the name of the page rendered in that language. What a treasure!
Example:
>>> gandhi.data['languages'][:3]
[{u'lang': u'af', u'title': u'Mahatma Gandhi'},
{u'lang': u'als', u'title': u'Mohandas Karamchand Gandhi'},
{u'lang': u'am', u'title': u'\u121b\u1205\u1270\u121b \u130b\u1295\u12f2'}]Methods: get_more(), get_querymore()
<str> lead section full HTML
This is the page's lead section, or summary, in full HTML including references, citations, and infoboxes.
Example:
gandhi.lead: <str(15770)> <span><p><b>Mohandas Karamchand Gandhi</b> (<sp...
Methods: get_restbase('/page/mobile-sections-lead')
<int> page length in bytes
This is the size of the page in bytes from Mediawiki API:Info.
Example:
>>> gandhi.data['length']
264127Methods: get_query()
<list> list of interwiki links
This is the list of interwiki links from the page's parsetree.
Example:
>>> gandhi.data['links'][:3]
[u'https://biblio.wiki/wiki/Mohandas_K._Gandhi',
u'https://commons.wikimedia.org/wiki/Special:Search/Mohandas_K._Gandhi',
u'https://en.wikiquote.org/wiki/Special:Search/Mohandas_K._Gandhi']Methods: get_parse()
<dict> last modified dates
This attribute contains the last modified dates of the page and its associated wikidata.
Example:
>>> gandhi.data['modified']
{'page': u'2017-09-23T17:28:59Z', 'wikidata': u'2017-09-23T16:39:57Z'}Methods: get_query(), get_restbase(), get_wikidata()
<int> Wikipedia database ID
This is the numeric identifier of the page in the Mediawiki database. It is useful as a pivot point for wptools to gather information across APIs.
Example:
>>> gandhi.data['pageid']
19379Methods: get_query(), get_parse(), get_restbase(), get_wikidata()
<str> page parsetree XML
This the full parsetree XML for the page which is used by wptools to parse infoboxes. It is certainly useful for a great many other things too.
Example:
Gandhi (en) data {
parsetree: <str(333213)> <root><template><title>Redirect</title><p...
}
Methods: get_parse()
<dict> page wikidata properties
This attribute contains any properties found in the page's wikidata. These properties are basically wikidata values. In wikibase, entities have claims (labels) and properties (values). Properties can have claims as values.
Example:
>>> gandhi.data['properties']
{u'P18': [u'Portrait Gandhi.jpg'],
u'P27': [u'Q129286', u'Q668'],
u'P31': [u'Q5'],
u'P345': [u'nm0003987'],
u'P569': [u'+1869-10-02T00:00:00Z'],
u'P570': [u'+1948-01-30T00:00:00Z'],
u'P910': [u'Q6512732']}Methods: get_wikidata()
<str> a random Mediawiki title
This attribute contains a random title that we get for free with some requests.
Example:
>>> gandhi.data['random']
u'Elfcon'Methods: get_query()
<str> the page's normalized title
This is the normalized title of the page from the APIs. Pretty straightforward!
Example:
>>> gandhi.data['title']
u'Mahatma Gandhi'Methods: get_parse(), get_query(), get_random(), get_restbase(), get_wikidata()
<str> canonical URL
This is the canonical URL formed from Mediawiki convention.
Example:
>>> gandhi.data['url']
u'https://en.wikipedia.org/wiki/Mahatma_Gandhi'Methods: get_query(), get_restbase()
<str> raw wikitext URL
This is the ostensible direct link to a page's wikitext. However, this link does not always resolve correctly, for instance, if there is a period in the title like 'J.R.R. Tolkien'.
Example:
>>> gandhi.data['url_raw']
u'https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw'Methods: get_query(), get_restbase()
<int> average daily page views
We average the daily page views from the last WEEK from API:Query prop=pageviews. No way!
Example:
>>> gandhi.data['views']
21718Methods: get_query(), get_querymore()
<int> number of page watchers
This is simply the number of people watching the page from Mediawiki API:Info. Intriguing!
Example:
>>> gandhi.data['watchers']
1733Methods: get_query()
<str> wikidata classification
This is Wikidata Property:P31 "instance of", which basically tells us something about what this page is. Incredibly useful if you're not familiar with the title and want to know what kind of data you are looking at.
Example:
>>> gandhi.data['what']
u'human'Methods: get_wikidata()
<str> wikibase item ID
This is the wikibase item identifier that represents an object, concept, or event in Wikidata.
Example:
>>> gandhi.data['wikibase']
u'Q1001'Methods: get_wikidata()
<dict> the actual wikidata for a page
This is the collection of wikidata that wptools has managed to gather for a page. Claim labels and properties that have claim values have been resolved into this attribute. As the Wikidata project matures, it will come closer to what we get with page.data['infobox'], but much better because it will have a standardized structure!
Example:
>>> gandhi.data['wikidata']
{'IMDB': u'nm0003987',
'birth': u'+1869-10-02T00:00:00Z',
'category': u'Category:Mahatma Gandhi',
'citizenship': [u'British Raj', u'India'],
'death': u'+1948-01-30T00:00:00Z',
'image': u'Portrait Gandhi.jpg',
'instance': u'human'}Methods: get_wikidata()
<str> wikidata URL
This is simply the URL to a page's Wikidata page.
Example:
>>> gandhi.data['wikidata_url']
u'https://www.wikidata.org/wiki/Q1001'Methods: get_parse(), get_query(), get_restbase(), get_wikidata()
<str> page wikitext
This is the raw wikitext used to render Mediawiki pages. It took me a while to figure out that there is absolutely no hope of reproducing the HTML that results from Mediawiki and its vast ecosystem of templates and add-ons from the raw wikitext yourself (see Parsoid). Phenomenal!
Example:
Gandhi (en) data
{
wikitext: <str(262663)> {{Redirect|Gandhi}}{{pp-move-indef}}{{pp...
}
Methods: get_parse()
Category (en) data { pageid, members, title }
<int> category page ID
The category page ID from API:Random.
Example:
>>> cat.data['pageid']
44375025Methods: wptools.category(), get_random()
<list> list of category members
This is the list of category members from API:Categorymembers.
Example:
>>> cat.data['members'][:3]
[{u'ns': 0, u'pageid': 43686772, u'title': u'The Jazz Messengers'},
{u'ns': 0, u'pageid': 10932853, u'title': u'Dale Barlow'},
{u'ns': 0, u'pageid': 32306397, u'title': u'Mickey Bass'}]<str> the category title
If you do not supply a title, a random category lookup will supply a title.
Example:
>>> cat = wptools.category()
en.wikipedia.org (random:14) 🍭
Category:Jazz Messengers (en) data
{
pageid: 44375025
title: Category:Jazz Messengers
}Methods: wptools.category(), get_random()
TBD