[Master-Issue] Issues with POST /samples + Training

[stopachka]

The Problem:

• There are many issues reported, where if a developer uses POST /samples, the training does not pick up these samples.
  • In addition to this, it is hard right now to for developers to get further clarity on what is happening with their app, as they do not have access to training information. They can't answer the question: "When did my app last train? Did it consider my samples?"

Attempted Repro:

• https://gist.github.com/stopachka/070f846384ce7a3d927093673711e995

The State on the Repro:

• We are currently not able to repro this

The Solution:

To kick this bugs bum, we need to do two things:

1. Help us with windowed bug reports

• If you experience these issues, please include in this thread:
  • Your App ID
  • The time window of when you sent these samples
  • The samples you were sending (you can email it to me at stopachka [at] fb [do.t] com
  • As much detail as you can about what you experience

2. We will introduce a new Training API

• To help give more clarity on this, we will introduce an API that will allow you to see when your app last trained, and when it is next scheduled to train. From the bug reprots, it's abundantly clear that this is needed, to build platforms on top of Wit.ai
  • I'll update here as we take next steps

[bpleao]

I have reproduced this behavior a number of times. All recent (last month) attempts I've made to train a wit bot with my data ends up in either not completing training (which is the focus of #879) or the behavior described here: response from the bot (using GET /message) is always empty, as if it did not consider the samples in training. If I use the console to check my bots, I can see all intents and associated samples there (I did not systematically checked if all of them are there, but could find all I've searched for). I'll describe more details here to try to help.

My app uses only the default intent (trait) entity. It's a Q&A system with 411 intents and a number of samples for each one. Recently I've been trying to train new apps with almost 13000 samples. I don't think the issue is with this architecture of the bot since in October I could successfully train apps with the same architecture and almost the double of this number of samples (although it frequently did not finish training and I had to try again). It's fully created using the HTTP API:

1. Create the app with POST /apps
2. Add intent values with PUT /entities/intent
3. Add samples with POST /samples, sending all samples for each intent value (which may be over 300) at a single call

Here are some app IDs in this situation:

Hope this helps.

[stopachka]

Hey @bpleao

I'll investigate the tokens you mentioned and let you know what I find. Thanks for the thorough report!

[rowanvanekeren]

hey i did some testing on app-id: 5a21524e-35e1-4ae9-b24c-2cd2092ee907
I know it's async but it still isnt working and i trained an intent hours ago. Like the intent new_haircut i added this to 'intent' with expression (in dutch) 'nieuw kapsel'. i also added other intents via the HTTP API the /samples and everything appears fine but no intent is found when i type 'nieuw kapsel' (new haircut) but it appears in at the intent expressions when i check the intent and it's value

[stopachka]

Hey @rowanvanekeren -- took a look at your app, the issue is that a sample made through the api, with an empty text string {text: ""}

• Fixing this now so that it does not happen again -- once that is done training should come back for your app. If you are in an urgent state, you can try to re-create a new app, but make sure on the client side that only samples with non-blank text strings are sent through.

all if you give me your app ids for more training issues, I'd highly appreciate it -- we are getting close to nabbing the bugs!

[rajasimon]

Training status API sounds interesting. And much needed for programmers to do wit.ai programmatically.

[l5t]

we are working ont it and hope to release it in the next few weeks...

[bpleao]

Hi, any news on this topic? Thanks

[patapizza]

Yes, we've made some progress on it.

[psquared2]

App ID 5a5f65cf-78ea-4382-b2a9-ea0a85c456d0

I'm training my app from API.

The payload from training keyword ("keyword") and trait ("topic") entity follows below - as you can see, the training data is successfully submitted but they are not showing up on the web interface.

Array ( 
[request] => POST /samples?v=20180121
HTTP/1.0 Authorization: Bearer xxx 
Content-Type: application/json 
User-Agent: Drupal (+http://drupal.org/) 
Host: api.wit.ai 
Content-Length: 242 
[{"text":"How do I apply for housing?","entities":[{"entity":"keyword","value":"apply","start":9,"end":14},{"entity":"keyword","value":"housing","start":19,"end":26},
{"entity":"keyword","value":"how"},
{"entity":"topic","value":"residence"}]}]
[data] => { "sent" : true, "n" : 1 }
[protocol] => HTTP/1.1
[status_message] => OK
[headers] => Array ( [server] => nginx [date] => Wed, 24 Jan 2018 18:25:07 GMT [content-type] => application/json [x-content-type-options] => nosniff [connection] => close [content-length] => 30 )
[code] => 200 )

Also, as I mentioned in #940, I cannot add metadata to trait entities (this is not a big deal for me personally, thought I would add it anyways).

[l5t]

Thanks for flagging, we do have a training lag at the moment so the queue will take some time to clear. I will update when back to normal

[bpleao]

I've tried to train again after some months but the behavior continues the same (stuck in training). App id 5a7b699e-81dd-4a8a-8889-085ddb127a3d
Any new advice based on the progress you're making on this topic? Thanks.

[psquared2]

@bpleao The quickest way is to email Wit.ai support to reset the app. It's very quick.

[Xzya]

Any update on this?
Could you also reset my app please ?
AppID: 5aa8de2f-d253-4747-83fd-51e387075683
Thank you!

[patapizza]

@Xzya Your app is now trained.

[gulbinas]

useful PR: wit-ai/pywit#101