Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #160: adding support to generate documentation from multiple repository in WEBUI #169

Merged
merged 1 commit into from
Jun 30, 2017

Conversation

poonai
Copy link
Contributor

@poonai poonai commented Jun 22, 2017

Description

I have added button to add the multiple input box for handling multiple repository

Related Issue

Motivation and Context

If project have more than one repository. it'll help to combine all the repository documentation under one umbrella

How Has This Been Tested?

Screenshots (if appropriate): generated documentation

multiple

Types of changes

  • Bug fix
  • New feature

Checklist:

  • I have read the Contribution & Best practices Guide and my PR follows them.
  • There is a corresponding issue for this pull request.
  • Mentioned the Issue number in the pull request commit message Fixes #<number> commit message
  • There is only one commit per issue.

}
});
return true;
// logic for one Repository
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sch00lb0y I know this is still a WIP, but I have a concern here. This is not a good design. The webUI shouldn't handle build logic. the purpose of the webUI should be to just take a list of subproject url's and pass them to generate.sh. In the event if there is no subproject pass a empty string. Handle the whole build logic in generate.sh. That way this would also work with travis out of the box. This is why some more discussion was needed in #160 before starting work on it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all the build process are happening in the bash script only, I just added a condition for a single repository or multiple repositories. We don't know how many features will be implemented in future. when we keep on adding thing to a single file, we making things much complex and also making hard to read as well as maintain. still we don't know building documentation for multiple repositories is our end goal. even i'm reusing old scripts like toc tree generation. we'll maintain separate script for WEBUI and travis. as discussed @imujjwal96 will work on Travis implementation

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me correct you there, @pri22296 is working on Travis implementation

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not saying add everything to generate.sh. You could create a different bash script and source it. That's not a problem. I don't understand why is the condition even needed? You could just have checked in the bash script whether SUBPROJECT_URLS is empty or not and if not empty clone them too. One of our primary goals should be that output remains same regardless of if the user used travis env vars, .yaydoc.yml or webUI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What i did when i was working on implementing multiple projects' script was, I added an if statement in the generate_ci.sh that checked if the SUBPROJECT_URLS is set or not. If it was set, it executed another script, otherwise it executed generate.sh

Copy link
Contributor

@imujjwal96 imujjwal96 Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pri22296 I have to disagree with you on this one. We cannot merge a PR on UI changes unless the backend script is working. So removing the script part from this PR would keep this PR on hold for a very long time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pri22296 Yeah sure I'll restrict this PR with UI. . If you already wrote the script. please make a PR I'll integrate with WEBUI. @imujjwal96 I just wrote only a few lines in build scripts and i'll scrape this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, how does this PR fixes partly #151 . I don't see any python test files

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry wrongly mentioned 😅 . I'll change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need @Sch00lb0y , that's already scrapped.

@poonai poonai changed the title [WIP] Fixes partly #151 adding support to generate documentation from multiple repository in WEBUI [WIP] Fixes partly #160 adding support to generate documentation from multiple repository in WEBUI Jun 23, 2017
@poonai poonai changed the title [WIP] Fixes partly #160 adding support to generate documentation from multiple repository in WEBUI Fixes partly #160 added button to add mutiple giturl input box Jun 23, 2017
@poonai poonai changed the title Fixes partly #160 added button to add mutiple giturl input box [WIP] Fixes partly #160 added button to add mutiple giturl input box Jun 23, 2017
@poonai poonai changed the title [WIP] Fixes partly #160 added button to add mutiple giturl input box Fixes #160: adding support to generate documentation from multiple repository in WEBUI Jun 27, 2017
@poonai poonai force-pushed the multiple branch 2 times, most recently from c1322b1 to 4bcab9f Compare June 27, 2017 17:52
@poonai
Copy link
Contributor Author

poonai commented Jun 27, 2017

@imujjwal96 @pri22296 Please review

@pri22296
Copy link
Member

@Sch00lb0y could you add a screenshot or maybe show a deployment?

@poonai
Copy link
Contributor Author

poonai commented Jun 28, 2017

@pri22296 demo deployment http://54.255.170.238/

@pri22296
Copy link
Member

@Sch00lb0y Here also you are using the first entry of the sub-projects as the giturl. don't do that. have a seperate field for giturl like it was before. and then have a seperate field for subprojects which would initially contain nothing and if users want they would add them. notice the word sub. The structure here is a tree. don't flatten it.

@poonai
Copy link
Contributor Author

poonai commented Jun 28, 2017

@pri22296 I'm not using the first URL of subproject as git_url. I'm taking as two entry as you mentioned in the previous PR #174. I added a button which add input box for the subproject and I'm joining with ',', passing it to script.

  var email = formData.email;
  var gitUrl = formData.gitUrl;
  var docTheme = formData.docTheme;
  var debug = formData.debug;
  var uniqueId = uuidV4();
  var webUI = "true";
  var subProject = ""
  if (formData.subProject != undefined) {
    subProject = formData.subProject.join(",")
  }

  var donePercent = 0;

  const args = [
    "-g", gitUrl,
    "-t", docTheme,
    "-m", email,
    "-d", debug,
    "-u", uniqueId,
    "-w", webUI,
    "-s", subProject
];

even i didn't touch the script. I'm just passing value to the script. if this is not the answer for your query. please elobrate your query

@pri22296
Copy link
Member

screenshot 165

So you are saying that the first entry after GITURL is well git_url, and the entries after that is subproject? If that is the case, could you add a proper label?

@poonai
Copy link
Contributor Author

poonai commented Jun 28, 2017

@pri22296 I have changed the lable please check

@pri22296
Copy link
Member

github api search is not working for subprojects as it works for giturl.

@poonai
Copy link
Contributor Author

poonai commented Jun 28, 2017

@pri22296 I made changes

@pri22296
Copy link
Member

Looks good. the initial suggestion when field is empty is still from the giturl. can that be fixed?

@poonai
Copy link
Contributor Author

poonai commented Jun 28, 2017

@pri22296 sorry, I'm not able to get you Please elaborate your query.

@pri22296
Copy link
Member

screenshot 166

suggestions from the first fields are being shown in the second field

@poonai
Copy link
Contributor Author

poonai commented Jun 28, 2017

@pri22296 I made this intentionally because it should wait for github api. if the same search query. It'll improve the experience

@poonai
Copy link
Contributor Author

poonai commented Jun 29, 2017

@imujjwal96 @pri22296 please review this

@pri22296
Copy link
Member

if the same search query. It'll improve the experience

Ok, it makes sense for the giturl field but Why would giturl and one of it's subproject will have the same query?

@poonai
Copy link
Contributor Author

poonai commented Jun 29, 2017

open-event-android
open-event-webapp.
There are projects look alike with few changes in suffix

@pri22296
Copy link
Member

Oh cool. nice job

@@ -90,7 +97,21 @@ function getData() {
if (field.name === "debug" ) { data.debug = field.value; }
if (field.name === "heroku_api_key" ) { data.herokuAPIKey = field.value.trim(); }
if (field.name === "heroku_app_name" ) { data.herokuAppName = field.value.trim(); }
if (field.name === "subproject_url[]") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is [] required in the name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To represent array I kept like this. if you want me to change then I'll change

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a standard notation for representing arrays? If it is, then it's fine, otherwise remove it. [] in a field name looks weird.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it is common to use

@imujjwal96
Copy link
Contributor

Allow empty values for sub project's url

@imujjwal96
Copy link
Contributor

imujjwal96 commented Jun 29, 2017

Failed to generate documentation
image

image

logs:

    Cloning into 'yaydoc_temp'...
Cloning into 'mydoc_temp'...
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: sphinx==1.5.5 in /usr/local/lib/python2.7/dist-packages (from -r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: recommonmark in /usr/local/lib/python2.7/dist-packages (from -r /home/ubuntu/yaydoc/requirements.txt (line 2))
Requirement already satisfied: pypandoc in /usr/local/lib/python2.7/dist-packages (from -r /home/ubuntu/yaydoc/requirements.txt (line 3))
Requirement already satisfied: pandocfilters in /usr/local/lib/python2.7/dist-packages (from -r /home/ubuntu/yaydoc/requirements.txt (line 4))
Requirement already satisfied: PyYaml in /usr/lib/python2.7/dist-packages (from -r /home/ubuntu/yaydoc/requirements.txt (line 5))
Requirement already satisfied: mock in /usr/local/lib/python2.7/dist-packages (from -r /home/ubuntu/yaydoc/requirements.txt (line 6))
Requirement already satisfied: Pygments>=2.0 in /usr/local/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: imagesize in /usr/local/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: babel!=2.0,>=1.3 in /usr/local/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: Jinja2>=2.3 in /usr/local/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: requests>=2.0.0 in /usr/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: docutils>=0.11 in /usr/local/lib/python2.7/dist-packages (from sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: commonmark<=0.5.4 in /usr/local/lib/python2.7/dist-packages (from recommonmark->-r /home/ubuntu/yaydoc/requirements.txt (line 2))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from pypandoc->-r /home/ubuntu/yaydoc/requirements.txt (line 3))
Requirement already satisfied: pip>=8.1.0 in /usr/local/lib/python2.7/dist-packages (from pypandoc->-r /home/ubuntu/yaydoc/requirements.txt (line 3))
Requirement already satisfied: wheel>=0.25.0 in /usr/local/lib/python2.7/dist-packages (from pypandoc->-r /home/ubuntu/yaydoc/requirements.txt (line 3))
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from mock->-r /home/ubuntu/yaydoc/requirements.txt (line 6))
Requirement already satisfied: pbr>=0.11 in /usr/local/lib/python2.7/dist-packages (from mock->-r /home/ubuntu/yaydoc/requirements.txt (line 6))
Requirement already satisfied: pytz>=0a in /usr/local/lib/python2.7/dist-packages (from babel!=2.0,>=1.3->sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python2.7/dist-packages (from Jinja2>=2.3->sphinx==1.5.5->-r /home/ubuntu/yaydoc/requirements.txt (line 1))
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning. SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning
Creating file ./conf.py. Creating file ./index.rst. Creating file ./Makefile. Creating file ./make.bat. Finished: An initial directory structure has been created. You should now populate your master file ./index.rst and create other documentation source files. Use the Makefile to build the docs, like so: make builder where "builder" is one of the supported builders, e.g. html, latex or linkcheck.
Running Sphinx v1.5.5
making output directory...
/usr/local/lib/python2.7/dist-packages/sphinx/util/requests.py:62: UserWarning: Some links may return broken results due to being unable to check the Server Name Indication (SNI) in the returned SSL cert against the hostname in the url requested. Recommended to install requests-2.4.1+. 'Some links may return broken results due to being unable to '
WARNING: while setting up extension conf.py: directive 'include' is already registered, it will be overridden
loading pickled environment...
not yet created
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 3 source files that are out of date
updating environment:
3 added, 0 changed, 0 removed
reading sources... [ 33%] index
reading sources... [ 66%] mydoc/index
reading sources... [100%] yaydoc/index
/home/ubuntu/yaydoc/temp/[email protected]/5edb5beb-b0be-46af-9c7b-f9bcc608794a/mydoc/index.rst:1: SEVERE: Problems with "mdinclude" directive path: InputError: [Errno 2] No such file or directory: 'README.md'.
/home/ubuntu/yaydoc/temp/[email protected]/5edb5beb-b0be-46af-9c7b-f9bcc608794a/yaydoc/index.rst:1: SEVERE: Problems with "mdinclude" directive path: InputError: [Errno 2] No such file or directory: 'README.md'.
looking for now-outdated files...
none found
pickling environment...
done
checking consistency...
done
preparing documents...
done
writing output... [ 33%] index
writing output... [ 66%] mydoc/index
writing output... [100%] yaydoc/index
/home/ubuntu/yaydoc/temp/[email protected]/5edb5beb-b0be-46af-9c7b-f9bcc608794a/index.rst:1: WARNING: toctree contains reference to document u'yaydoc/index' that doesn't have a title: no link will be generated
/home/ubuntu/yaydoc/temp/[email protected]/5edb5beb-b0be-46af-9c7b-f9bcc608794a/index.rst:7: WARNING: toctree contains reference to document u'mydoc/index' that doesn't have a title: no link will be generated
generating indices...
genindex
writing additional pages...
search
copying static files...
done
copying extra files...
done
dumping search index in English (code: en) ...
done
dumping object inventory...
done
build succeeded, 5 warnings.
Build finished. The HTML pages are in _build/html. 

@pri22296
Copy link
Member

@imujjwal96 weird. coz i tried generating it and got a fine result
http://54.255.170.238/preview/[email protected]/543438d9-ca50-4761-b87e-6e50bc005c91_preview/

@imujjwal96
Copy link
Contributor

You have the input that I provided and the logs that it generated.
Am I not aware of certain caveats in inputting the form fields?

@pri22296
Copy link
Member

I tried it with other repos. With your input, I'm also getting a blank page. Currently there are many issues with multiple_generate.sh. I will be trying to sort all of them in my PR for #196 . In the time being, maybe just review this PR for UI only?

@poonai
Copy link
Contributor Author

poonai commented Jun 29, 2017

@imujjwal96 because yaydoc having index.rst in the docs folder which is causing the problem. This is get solved as soon as #199 get merged

@pri22296
Copy link
Member

pri22296 commented Jun 29, 2017

@Sch00lb0y I will have to disagree with you on this one. The example with which i tried also had index.rst in the docs folder.
screenshot 168

@poonai
Copy link
Contributor Author

poonai commented Jun 29, 2017

@pri22296 I think none of the repository index.rst is linking the file which is outside the docs folder. because in multiple_generate.sh we deleting everything other than docs folder

@poonai
Copy link
Contributor Author

poonai commented Jun 29, 2017

@imujjwal96

Allow empty values for sub project's url

made changes for this

@@ -29,14 +30,20 @@ $(function () {

socket.on('logs', function (data) {
$('#messages').append($('<li class="info">').text(data.data));
$("#progress").css("width", data.donePercent + "%");
if (data.donePercent >= 100) {
$("#progress").css("width", "90%");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't this mean that if progress goes from 95 to 100, the bar would actually roll back to 90?

Copy link
Contributor Author

@poonai poonai Jun 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it won't go between 90 to 100.some time multiple_generate.sh will execute some time generate.sh will execute. we don't know how many logs will emit in future so, I made a tap that I should not go above 90 before execution of exit code 0.

Copy link
Contributor

@imujjwal96 imujjwal96 Jun 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be (data.donePercent >= 90)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made changes @imujjwal96 Thanks

@mariobehling mariobehling merged commit 898665d into fossasia:master Jun 30, 2017
@poonai poonai deleted the multiple branch July 14, 2017 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants