Explore big dataset disk_stories_full in mongo db #197
Comments
|
77300000 finished processing Mongo db just died in the middle not sure why!!! |
|
Try and use the |
|
By mongo file do u mean access that table, it is but directly access the file on disk, it get the data using pumibgo through mongo db |
|
*it is not directly |
|
@YanLiang1102 by mongo document, I mean the MongoClient. sleep = 1
done = False
while not done:
try:
# your code HERE
done = True
except pymongo.AutoReconnect:
logging.info("Error connecting sleeping for {}".format(pow(2, sleep)))
time.sleep(pow(2, sleep))
sleep += 1
logging.info("retrying...") |
|
So u suspect our network will time out when the connection is keeping alive for too long like 3 or 4 hours, I kind of think it is mongo client issue , they might only to keep the db connection for a certain amount of time @cegme |
|
The network can go out and interrupt a TCP at any time. A local run should have a network problem still adding a sleep can be a remedy. Add |
|
@cegme I don't think the code will work Dr Grant, in the way you write the cursor is changed, so it will loop from the beginning. |
|
File "1021.py", line 20, in During handling of the above exception, another exception occurred: Traceback (most recent call last): NameError: name 'pymongo' is not defined |
|
put limit on poolSize does help , but the mongo demon will down too, |
…on problem, since teh previous implemenation when redo teh cursor will restart again so it will next ends
using c++ directly plan to to gain the efficiency,
here is one record like a schema in the table:
{ "_id" : ObjectId("572c0953172ab83173aaf011"), "news_source" : "Associated Press International", "position_section" : "SPORTS NEWS", "word_count" : "577", "states" : [ "OHIO, USA" ], "id_type" : "DOC-ID", "date_added" : ISODate("2016-05-06T03:02:43.316Z"), "cities" : [ "OAKLAND, CA, USA" ], "article_title" : "Raiders go QB route for Fresno State's Carr", "article_body" : "NEW YORK (AP) - Derek Carr, the brother of former No.1 draft pick David Carr, was selected by the Oakland Raiders with selection 36 of the 2014 edition on Friday. A day after the first-round selections were completed, teams went about filling their rosters with further selections on Friday. Derek Carr is a quarterback like his brother, went to Fresno State, like his brother, and enters the league with a wife and child, like his brother, but is hoping the similarities end there, given David Carr's disappointing career after being drafted by Houston with much fanfare back in 2002. "I learned everything that he did right and everything that he did wrong," Derek Carr said. "He told me that if he could do anything, he hopes he made the path smoother for me as I transition into the NFL." Derek Carr rewrote the record book in his time in college, throwing for more than 10,000 yards and 100 touchdown passes, leading Fresno State to consecutive Mountain West Conference titles. Oakland has veteran Matt Schaub earmarked to be its starting quarterback, but he will get a serious push from Carr. In other picks Friday: - Houston used the first pick of the second round on UCLA guard Xavier Su'a-Filo, who joins the first overall pick, Jadeveon Clowney, in a upgraded defensive line. The 6-foot-4, 307-pound Su'a-Filo, who went on a Mormon mission while in college, also has played tackle. - The Cowboys took Boise State defensive end Demarcus Lawrence, who they hope will emulate their departed sacks leader with the same first name, DeMarcus Ware, now with Denver. "I'm my own Demarcus," Lawrence said. "I don't like to try to be nobody else. I'm going to be me, and I'm going to do it well." - Cleveland added a protector for new quarterback Johnny Manziel by grabbing guard Joel Bitonio of Nevada, who also can play tackle or center. The Browns caused the biggest stir on opening night when they traded up to No. 22 to get 'Johnny Football'. "He's a heck of a quarterback," Bitonio said. "Hopefully, he comes in and he's ready to compete and just ready to work and do well for the Cleveland Browns." Cleveland did not choose any receivers even though Josh Gordon is reportedly facing suspension by the NFL for violating the league's drug policy again. Gordon was suspended for the first two games of 2013, but still led the league with 1,646 yards receiving. - Eastern Illinois quarterback Jimmy Garoppolo went to New England near the end of the second round, and will be a backup to his favorite player Tom Brady. "Whether I was coming in as the starter or as the backup, I'm going to go in and approach it the same way," Garoppolo said. "I'm going to go out there and try to get better each and every day. That's what good football players do." - Washington may have got a bargain by selecting Virginia tackle Morgan Moses at pick 66. Moses had been earmarked as a potential first-round pick by many pundits but never received a call. "I thought my phone was broken," Moses quipped. It took 54 selections, a draft record, for a running back to go. Bishop Sankey of Washington was chosen by Tennessee, who cut Chris Johnson this spring. Two more went in the next three selections: Jeremy Hill of Louisiana State to Cincinnati, and Carlos Hyde of Ohio State to San Francisco. ___ AP College Football Writer Ralph D. Russo and Sports Writers Simmi Buttar, Schuyler Dixon and Josh Dubow contributed to this story. ___ AP NFL website: www.pro32.ap.org and www.twitter.com/AP_NFL", "language" : "english", "stanford" : 0, "countries" : [ "UNITED STATES" ], "publication_date_raw" : "May 10, 2014 Saturday", "doc_id" : "TOPNEWSa6f3d910954c4d228015df686134c54d", "parsed" : 1, "queue_added" : 0 }
The text was updated successfully, but these errors were encountered: