Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chatbot parser output linux tutorial #679

Draft
wants to merge 153 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
1ebc363
initial commit
EwDa291 Aug 8, 2024
34df842
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 8, 2024
10edb20
some cleanup
EwDa291 Aug 8, 2024
85a93ec
used jinja to replace macros
EwDa291 Aug 9, 2024
dfff5fa
adapt if-mangler to accommodate for nested if-clauses
EwDa291 Aug 9, 2024
649ddec
adapt the parser to take all files as input, not all files get parsed…
EwDa291 Aug 9, 2024
2116d6e
adapt the parser to take all files as input, not all files get parsed…
EwDa291 Aug 9, 2024
159aa62
small update, not important
EwDa291 Aug 9, 2024
75765e5
change to the templates
EwDa291 Aug 9, 2024
57d9cfe
change to accommodate for more nested if-clauses
EwDa291 Aug 9, 2024
75d345b
Delete scripts/HPC chatbot preprocessor/start_checker.py
EwDa291 Aug 9, 2024
ff7a9fc
make sure files with duplicate names between normal files and linux-t…
EwDa291 Aug 12, 2024
47a33b7
Merge branch 'chatbot_parser' of https://github.com/EwDa291/vsc_user_…
EwDa291 Aug 12, 2024
7d279d6
fixed the problem of some files being written in reST instead of mark…
EwDa291 Aug 12, 2024
8047572
some small fixes
EwDa291 Aug 12, 2024
7d1c5ed
remove try-except-structure
EwDa291 Aug 13, 2024
984b0cd
collapse all code into one file
EwDa291 Aug 13, 2024
8f5eeaa
Rename file
EwDa291 Aug 13, 2024
2b97b7a
cleanup repository
EwDa291 Aug 13, 2024
b595301
Rename directory
EwDa291 Aug 13, 2024
90c8ab7
add a main function
EwDa291 Aug 13, 2024
b8ae706
make file paths non os-specific
EwDa291 Aug 13, 2024
b751497
use docstrings to document the functions
EwDa291 Aug 13, 2024
0f8eb5d
rewrite the if-mangler to make it more readable
EwDa291 Aug 13, 2024
9938e92
got rid of most global variables
EwDa291 Aug 13, 2024
508b22c
fixed some issues with if statements
EwDa291 Aug 13, 2024
a25ce2d
fixed some issues with if statements
EwDa291 Aug 13, 2024
80d0535
got rid of all global variables
EwDa291 Aug 13, 2024
9163a75
small changes to make file more readable
EwDa291 Aug 14, 2024
1dcffc1
codeblocks, tips, warnings and info reformatted
EwDa291 Aug 14, 2024
4d7fbdb
small optimisations
EwDa291 Aug 14, 2024
671f7f3
small optimisations
EwDa291 Aug 14, 2024
e5c39bd
initial commit
EwDa291 Aug 14, 2024
c6492fc
added requirements
EwDa291 Aug 14, 2024
aff8198
added requirements and usage info
EwDa291 Aug 14, 2024
a981002
minor changes to the print statements
EwDa291 Aug 14, 2024
1f3b343
reworked function to take care of html structures
EwDa291 Aug 16, 2024
b6388d3
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 16, 2024
48cad97
filter out images
EwDa291 Aug 16, 2024
df58f23
get rid of backquotes, asterisks, pluses and underscores used for for…
EwDa291 Aug 16, 2024
c423e07
dump to json files instead of txt files
EwDa291 Aug 16, 2024
2c333fe
cleaned up parser with macros
EwDa291 Aug 16, 2024
ce52352
cleaned up parser with macros
EwDa291 Aug 16, 2024
5db34af
cleaned up parser with macros
EwDa291 Aug 16, 2024
4226d28
Update README.md
EwDa291 Aug 19, 2024
d730a26
Update README.md
EwDa291 Aug 19, 2024
f3182e3
added section about restrictions on input files
EwDa291 Aug 19, 2024
aee54de
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 19, 2024
675bec5
adapted section about restrictions on input files
EwDa291 Aug 19, 2024
f1e58ef
adapted section about restrictions on input files
EwDa291 Aug 19, 2024
2bf1075
Merge branch 'chatbot_parser' of https://github.com/EwDa291/vsc_user_…
EwDa291 Aug 19, 2024
a168509
change variables to be lowercase
EwDa291 Aug 19, 2024
09b86c9
take out some copy pasting
EwDa291 Aug 19, 2024
f95b99e
added warning about long filepaths
EwDa291 Aug 19, 2024
06bb7b9
fixing typos
EwDa291 Aug 19, 2024
2f3e5b3
take out copy pasting
EwDa291 Aug 19, 2024
0c4dbe8
first draft version of the restructured script to accommodate for the…
EwDa291 Aug 20, 2024
38c4572
added support to filter out collapsable admonitions
EwDa291 Aug 20, 2024
5cbd653
attempt at fix for problems with jinja include, not working yet
EwDa291 Aug 20, 2024
0e6f8b2
fixed an issue with jinja templates
EwDa291 Aug 21, 2024
cd77837
added docstrings to new functions
EwDa291 Aug 21, 2024
98eb695
only add necessary if-statements in front of non-if-complete sections
EwDa291 Aug 21, 2024
27457e3
fixed some more jinja problems
EwDa291 Aug 21, 2024
bb72287
implemented extra test to make sure generic files dont accidentally g…
EwDa291 Aug 21, 2024
67cb19e
make sure empty os-specific files are not saved
EwDa291 Aug 21, 2024
cf9834a
clean up unused code
EwDa291 Aug 21, 2024
da32459
introduce more macros
EwDa291 Aug 21, 2024
093200b
reintroduce logic to remove unnecessary directories
EwDa291 Aug 21, 2024
5d0ffe9
added functionality to include links or leave them out
EwDa291 Aug 21, 2024
a3e34a9
added functionality to include links or leave them out
EwDa291 Aug 21, 2024
7c6154b
adapt filenames to allow for splitting on something other than subtitles
EwDa291 Aug 21, 2024
8d5b50d
making some changes to prepare to add paragraph level splitting tomorrow
EwDa291 Aug 21, 2024
0c10376
making some changes to prepare to add paragraph level splitting tomorrow
EwDa291 Aug 21, 2024
f8ee860
making some changes to prepare to add paragraph level splitting tomorrow
EwDa291 Aug 21, 2024
6533733
adapted the parsing script to allow for testing in a semi-efficient way
EwDa291 Aug 21, 2024
2e7a00f
added test for make_valid_title
EwDa291 Aug 21, 2024
f5e0579
removed useless lines from testscript
EwDa291 Aug 21, 2024
6757b4f
First attempt at splitting in paragraphs (need for other fixes for ti…
EwDa291 Aug 22, 2024
6d9558d
make two functions for different ways of dividing the text
EwDa291 Aug 22, 2024
2c7025a
added docstrings to new functions
EwDa291 Aug 22, 2024
ae99bb9
update test for valid titles
EwDa291 Aug 22, 2024
084b421
fixed problem with splitting os-specific text (metadata not fixed yet)
EwDa291 Aug 22, 2024
cf7f5f0
fix for metadata of os-specific sections
EwDa291 Aug 22, 2024
b7c10d3
clean up temporary version
EwDa291 Aug 22, 2024
4a441f3
added command line options for custom macros
EwDa291 Aug 22, 2024
662134f
small fix to macros
EwDa291 Aug 22, 2024
05eab4a
clean up test for valid title
EwDa291 Aug 22, 2024
b85a8fb
add a test for write_metadata
EwDa291 Aug 22, 2024
39a3c99
added functionality to split on paragraphs
EwDa291 Aug 23, 2024
af9e6cc
clean up
EwDa291 Aug 23, 2024
f4163a7
clean up
EwDa291 Aug 23, 2024
833f964
further clean up and added shebang
EwDa291 Aug 23, 2024
79b1a56
clean up
EwDa291 Aug 23, 2024
cec154c
added test for if mangler
EwDa291 Aug 23, 2024
2f4a277
clean up
EwDa291 Aug 23, 2024
cd0c8eb
clean up customizable options
EwDa291 Aug 23, 2024
3be262a
further adapt the script to be able to test it
EwDa291 Aug 26, 2024
1d32aab
make changes to usage in command line to be more intuitive
EwDa291 Aug 26, 2024
5902c96
first revised version of the README
EwDa291 Aug 26, 2024
6f97d5f
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 26, 2024
6e48800
added docstring to main function
EwDa291 Aug 26, 2024
0bc440b
include chatbot_prepprocessor
EwDa291 Aug 26, 2024
e6e6023
added options for source and destination directories
EwDa291 Aug 26, 2024
a6d99d9
cleanup
EwDa291 Aug 26, 2024
2be834f
cleanup
EwDa291 Aug 26, 2024
532543a
cleanup
EwDa291 Aug 26, 2024
107464e
relocate test files
EwDa291 Aug 26, 2024
dd64381
update arguments of if mangler
EwDa291 Aug 26, 2024
ef3fd58
relocate full test files
EwDa291 Aug 26, 2024
4d7db8f
Revert "update arguments of if mangler"
EwDa291 Aug 26, 2024
df9bac5
Revert "relocate full test files"
EwDa291 Aug 26, 2024
631d9e9
update test to adapt to new arguments in if mangler
EwDa291 Aug 26, 2024
c6e600d
relocated full test files
EwDa291 Aug 26, 2024
d1c6194
Rename test_paragraph_split_1.md to test_paragraph_split_1_input.md
EwDa291 Aug 26, 2024
695ffd6
Rename test_title_split_1.md to test_title_split_1_input.md
EwDa291 Aug 26, 2024
af4832b
smal fix
EwDa291 Aug 26, 2024
8805c8c
test text for paragraph split
EwDa291 Aug 26, 2024
a265ffd
start of a fix for double title problem, not done yet
EwDa291 Aug 26, 2024
6c2a61c
Fix for double title bug when splitting on paragraph
EwDa291 Aug 27, 2024
ed08879
Fix bug for empty linklist in metadata
EwDa291 Aug 27, 2024
176af13
fix bug where too many directories were sometimes created
EwDa291 Aug 27, 2024
d4ceac8
test of full script, test files not ready to be pushed yet
EwDa291 Aug 27, 2024
815a863
updated requirements.txt
EwDa291 Aug 27, 2024
d15469f
updated docstring in main function
EwDa291 Aug 27, 2024
daa6b36
add support for comments for the bot to be included in the source files
EwDa291 Aug 27, 2024
4c19f44
changed the default for min paragraph length
EwDa291 Aug 27, 2024
9a6ff58
added test files for full script test
EwDa291 Aug 27, 2024
56543f0
small fix for double title bug
EwDa291 Aug 27, 2024
52a3861
added examples of output of the script when splitting on paragraphs w…
EwDa291 Aug 27, 2024
692e77b
fix for issue with html links
EwDa291 Aug 27, 2024
7f493a1
fix for issue with html links
EwDa291 Aug 27, 2024
0e34396
fix for issue with relative links to the same document
EwDa291 Aug 27, 2024
fa00044
added test for replace_markdown_markers
EwDa291 Aug 27, 2024
b3952b2
fix to small inconsistency in metadata
EwDa291 Aug 27, 2024
73072bf
added test for insert_links
EwDa291 Aug 27, 2024
3161309
make sure paragraphs only include full lists
EwDa291 Aug 28, 2024
7d4d7f9
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 28, 2024
3407be3
adapted to the new source files
EwDa291 Aug 28, 2024
6d04bbc
add source-directory to metadata and verbose mode
EwDa291 Aug 28, 2024
f33cfb3
added verbose mode
EwDa291 Aug 28, 2024
1c389d7
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 28, 2024
3227f19
Added limitation on lists
EwDa291 Aug 29, 2024
67aed53
fix for non os-specific if-statement not being recognised
EwDa291 Aug 29, 2024
9e297b1
new test for links
EwDa291 Aug 29, 2024
b6b8610
new test to make sure lists are kept as one section
EwDa291 Aug 29, 2024
57a2139
updated test_file for list test
EwDa291 Aug 29, 2024
170a10c
dropped <> around links and started new function to calculate length …
EwDa291 Aug 30, 2024
04efff6
removed parsed mds
EwDa291 Aug 30, 2024
1ef1f10
Changed paragraphs to decide length based on tokens instead of charac…
EwDa291 Aug 30, 2024
621c0a3
Changed paragraphs to decide length based on tokens instead of charac…
EwDa291 Aug 30, 2024
adf364d
Changed paragraphs to decide length based on tokens instead of charac…
EwDa291 Aug 30, 2024
e36482e
Added output of chatbot parser script (linux tutorial)
EwDa291 Aug 30, 2024
8f54b86
removing unnecessary files
EwDa291 Aug 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
new test for links
  • Loading branch information
EwDa291 committed Aug 29, 2024
commit 9e297b18ef9827a20a1283053ad49c3e081044e7
6 changes: 5 additions & 1 deletion scripts/HPC_chatbot_preprocessor/README.md
Original file line number Diff line number Diff line change
@@ -172,7 +172,7 @@ Any comments within the markdown files (for example TODO's) should follow the fo
Comments can be written in such a way that the script will keep them as input for the bot. To do that, the marker `INPUT_FOR_BOT` should be put in front of the content of the comment as such.

```
<!--INPUT_FOR_BOTyour comment for the bot-->
<!--INPUT_FOR_BOT: your comment for the bot-->
```

This will be reworked to
@@ -190,3 +190,7 @@ Due to the nature of this script, it can generate large directories with very lo
### Markdown lists

The parser is made in a way to detect lists and not split them in multiple paragraphs. The kinds of lists it can detect is all lists with denominators `-`, `+`, `*` and list indexed with numbers or letters (one letter per list entry). It can handle list entries being spread out over multiple lines if there is an indentation of at least two spaces. It can also handle multiple paragraph list entries in this way, as long as the indentation stays.

### Links

Part of the metadata of the parser are links. In order for the links to be built up in the right way, links to external sites should always start with either `https://` or `http://`.
89 changes: 65 additions & 24 deletions scripts/HPC_chatbot_preprocessor/chatbot_parser.py
Original file line number Diff line number Diff line change
@@ -104,7 +104,7 @@
METADATA_EXTENSION = "_metadata"

# Marker for comments for the bot
INPUT_FOR_BOT = "INPUT_FOR_BOT"
INPUT_FOR_BOT = "INPUT_FOR_BOT: "

# Standard strings for verbose output
LINE = "------------------------------------------------------------------------------------------------------\n"
@@ -138,34 +138,69 @@ def check_for_title(line, in_code_block, curr_dirs, options):
return 0


def replace_markdown_markers(curr_line, linklist, in_code_block, main_title):
def make_valid_link(link, main_title, is_linux_tutorial):
"""
Function that converts a string to a valid link to be used in the metadata

:param link: the input string to be turned into a valid link
:param main_title: the main title of the file that contains the link
:param is_linux_tutorial: boolean indicating whether the current file is part of the linux tutorial
:return link: the valid link
"""

# ugly fix for problem with links
linux_tutorial_files = ["beyond_the_basics", "common_pitfalls", "getting_started", "hpc_infrastructure", "index", "manipulating_files_and_directories", "navigating", "uploading_files"]
if is_linux_tutorial and any([linux_tutorial_files[i] in link for i in range(len(linux_tutorial_files))]):
linux_part = LINUX_TUTORIAL + '/'
else:
linux_part = ""

if link.startswith('http://') or link.startswith('https://') or link.startswith('mailto:'):
pass
else:
if link.startswith("./"):
link = link.replace('./', '')
elif link.startswith("../"):
link = link.replace('../', '')

if link.startswith("#"):
link = DOCS_URL + '/' + linux_part + main_title + "/" + link
elif link.endswith(".md") and ("/" not in link or "." not in link.split("/")[0]):
link = DOCS_URL + '/' + linux_part + link.replace(".md", "")
elif '.md#' in link:
link = DOCS_URL + '/' + linux_part + link.replace(".md", "/")
else:
link = DOCS_URL + '/' + linux_part + link

link = link.replace('index/', '').replace('/index', '')

return link


def replace_markdown_markers(curr_line, linklist, in_code_block, main_title, is_linux_tutorial):
"""
function that replaces certain markdown structures with the equivalent used on the website

:param curr_line: the current line on which markdown structures need to be replaced
:param linklist: the list used to store links that need to be printed at the end of the file
:param in_code_block: boolean indicating whether the current line is part of a code block
:param main_title: the main title of the file that is being processed
:param is_linux_tutorial: boolean indicating whether the current file is part of the linux tutorial
:return curr_line: the adapted current line
:return linklist: the updated linklist
"""

# replace images with an empty line
if re.search(r'(?i)!\[image]\(.*?\)', curr_line) or re.search(r'!\[]\(img/.*?.png\)', curr_line):
if re.search(r'(?i)!\[image]\(.*?\)', curr_line) or re.search(r'!\[.*?]\(img/.*?\.png\)', curr_line):
curr_line = ""

# replace links with a reference
matches = re.findall(r'\[(.*?)]\((.*?)\)', curr_line)
if matches:
for match in matches:
curr_line = curr_line.replace(f"[{match[0]}]({match[1]})", match[0] + LINK_MARKER + str(len(linklist)) + LINK_MARKER)
if ".md" not in match[1]:
if "#" not in match[1]:
linklist.append(match[1])
else:
linklist.append(DOCS_URL + "/" + main_title.replace(".md", "") + "/" + match[1])
else:
linklist.append(DOCS_URL + "/" + match[1].replace(".md", "/").replace("index", "").rstrip("/"))

linklist.append(make_valid_link(match[1], main_title, is_linux_tutorial))

# codeblock (with ``` -> always stands on a separate line, so line can be dropped)
if '```' in curr_line:
@@ -238,13 +273,14 @@ def replace_markdown_markers(curr_line, linklist, in_code_block, main_title):
return curr_line, linklist


def split_text(file, main_title, options, current_paragraph_number=-1, OS=GENERIC):
def split_text(file, main_title, options, is_linux_tutorial, current_paragraph_number=-1, OS=GENERIC):
"""
Function that splits the text into smaller sections and makes them into two dictionaries containing text and metadata

:param file: the filepath of the file to be split
:param main_title: the main title of the file
:param options: dictionary containing the options given by the user
:param is_linux_tutorial: boolean indicating whether the current file is part of the linux tutorial
:param current_paragraph_number: number of the paragraph that is being split, only applicable when splitting an os-specific paragraph on paragraph level
:param OS: the OS of the file to be split, only applicable when splitting an os-specific paragraph on paragraph level
:return paragraphs_text: dictionary containing the split sections of text
@@ -253,18 +289,19 @@ def split_text(file, main_title, options, current_paragraph_number=-1, OS=GENERI
"""

if options[SPLIT_ON_TITLES]:
return split_on_titles(file, main_title, options)
return split_on_titles(file, main_title, options, is_linux_tutorial)
elif options[SPLIT_ON_PARAGRAPHS]:
return split_on_paragraphs(file, main_title, options, current_paragraph_number, OS)
return split_on_paragraphs(file, main_title, options, is_linux_tutorial, current_paragraph_number, OS)


def split_on_titles(file, main_title, options):
def split_on_titles(file, main_title, options, is_linux_tutorial):
"""
Function that splits the text into smaller sections based on the subtitle structure and makes them into two dictionaries containing text and metadata

:param file: the filepath of the file to be split
:param main_title: the main title of the file
:param options: dictionary containing the options given by the user
:param is_linux_tutorial: boolean indicating whether the current file is part of the linux tutorial
:return paragraphs_text: dictionary containing the split sections of text
:return paragraphs_metadata: dictionary containing the metadata of each split section of text
:return subtitle_order: list containing all encountered subtitles in order of appearance
@@ -356,7 +393,7 @@ def split_on_titles(file, main_title, options):

# line is not a title
elif after_first_title:
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title)
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title, is_linux_tutorial)
if line != "\n":
current_paragraph += line

@@ -366,7 +403,7 @@ def split_on_titles(file, main_title, options):
last_dir = curr_dirs[last_title_level]
else:
previous_contained_if = True
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title)
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title, is_linux_tutorial)
if line != "\n":
current_paragraph += line

@@ -384,13 +421,14 @@ def split_on_titles(file, main_title, options):
return paragraphs_os_text, paragraphs_os_free_text, paragraphs_metadata, subtitle_order


def split_on_paragraphs(file, main_title, options, current_paragraph_number=-1, OS=GENERIC):
def split_on_paragraphs(file, main_title, options, is_linux_tutorial, current_paragraph_number=-1, OS=GENERIC):
"""
Function that splits the text into smaller sections based on the paragraph structure and makes them into two dictionaries containing text and metadata

:param file: the filepath of the file to be split
:param main_title: the main title of the file
:param options: dictionary containing the options given by the user
:param is_linux_tutorial: boolean indicating whether the current file is part of the linux tutorial
:param current_paragraph_number: number of the paragraph that is being split, only applicable when splitting an os-specific paragraph
:param OS: the OS of the file to be split, only applicable when splitting an os-specific paragraph
:return paragraphs_text: dictionary containing the split sections of text
@@ -524,12 +562,12 @@ def split_on_paragraphs(file, main_title, options, current_paragraph_number=-1,
# make a new title
metadata_title = make_valid_title(line[title_level + 1:-1])

line, link_list = replace_markdown_markers(line[title_level + 1:], link_list, in_code_block, main_title)
line, link_list = replace_markdown_markers(line[title_level + 1:], link_list, in_code_block, main_title, is_linux_tutorial)
current_paragraph += line

# line is not a title or the beginning of a new paragraph
elif line != "\n" or previous_contained_if:
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title)
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title, is_linux_tutorial)
current_paragraph += line

# keep track of title level and directory to write to metadata upon discovering a new subtitle
@@ -538,7 +576,7 @@ def split_on_paragraphs(file, main_title, options, current_paragraph_number=-1,
last_dir = curr_dirs[last_title_level]
else:
previous_contained_if = True
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title)
line, link_list = replace_markdown_markers(line, link_list, in_code_block, main_title, is_linux_tutorial)
current_paragraph += line

# create a title for the last paragraph
@@ -799,7 +837,7 @@ def make_valid_title(title):
valid_filename = re.sub(invalid_chars, '', title)

# Strip leading/trailing whitespace
valid_filename = valid_filename.strip().strip('-').replace(' ', '-')
valid_filename = valid_filename.strip().strip('-').replace(' ', '-').replace("--", "-")

return valid_filename

@@ -889,7 +927,10 @@ def write_files(title, text, paragraphs_metadata, title_order, title_order_numbe
os_part = ""
else:
os_part = LINK_OS[OS] + "/"
metadata[REFERENCE_LINK] = DOCS_URL + "/" + os_part + linux_part + paragraphs_metadata[title][MAIN_TITLE] + "/#" + ''.join(char.lower() for char in paragraphs_metadata[title][SUBTITLE] if char.isalnum() or char == '-').strip('-')
if "index" not in paragraphs_metadata[title][MAIN_TITLE]:
metadata[REFERENCE_LINK] = DOCS_URL + "/" + os_part + linux_part + paragraphs_metadata[title][MAIN_TITLE] + "/#" + ''.join(char.lower() for char in paragraphs_metadata[title][SUBTITLE] if char.isalnum() or char == '-').strip('-')
else:
metadata[REFERENCE_LINK] = DOCS_URL

# write metadata to file
with open(os.path.join(filepath, file_title + METADATA_EXTENSION + ".json"), 'w') as writefile:
@@ -964,7 +1005,7 @@ def split_and_write_os_specific_section(text, metadata, subtitle_order, title_or
writefile.write(jinja_text)

# split in right way
_, os_specific_text, os_specific_metadata, os_subtitle_order = split_text(TEMP_JINJA_FILE, metadata[MAIN_TITLE], options, current_paragraph_number=subtitle_order[title_order_number].split('_')[-1], OS=OS)
_, os_specific_text, os_specific_metadata, os_subtitle_order = split_text(TEMP_JINJA_FILE, metadata[MAIN_TITLE], options, is_linux_tutorial, current_paragraph_number=subtitle_order[title_order_number].split('_')[-1], OS=OS)

# prepare variables to fix metadata
total_subtitle_order = subtitle_order[:title_order_number] + os_subtitle_order + subtitle_order[title_order_number+1:]
@@ -1110,7 +1151,7 @@ def main(options):
print("\nSplitting the file for the first time (split in sufficiently small generic sections and large os-specific chunks)")

# split the text in paragraphs
paragraphs_os_text, paragraphs_os_free_text, paragraphs_metadata, subtitle_order = split_text(copy_file, main_title, options)
paragraphs_os_text, paragraphs_os_free_text, paragraphs_metadata, subtitle_order = split_text(copy_file, main_title, options, is_linux_tutorial)

if options[VERBOSE]:
print("\nFurther splitting os-specific chunks and writing generic and os-specific sections to files with metadata")
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
"title_depth": 2,
"directory": "account",
"links": {
"0": "../sites/hpc_policies"
"0": "https://docs.hpc.ugent.be/sites/hpc_policies"
},
"parent_title": "",
"previous_title": null,
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
"title_depth": 2,
"directory": "connecting",
"links": {
"0": "https://docs.hpc.ugent.be/connecting/../linux-tutorial/uploading_files/#copying-faster-with-rsync"
"0": "https://docs.hpc.ugent.be/linux-tutorial/uploading_files/#copying-faster-with-rsync"
},
"parent_title": "",
"previous_title": "connecting_paragraph_14",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "connecting",
"links": {
"0": "https://docs.hpc.ugent.be/web_portal",
"1": "https://docs.hpc.ugent.be/connecting/../troubleshooting/#issues-connecting-to-login-node"
"1": "https://docs.hpc.ugent.be/troubleshooting/#issues-connecting-to-login-node"
},
"parent_title": "",
"previous_title": "connecting_paragraph_2",
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
"title_depth": 2,
"directory": "connecting",
"links": {
"0": "../useful_linux_commands"
"0": "https://docs.hpc.ugent.be/useful_linux_commands"
},
"parent_title": "",
"previous_title": "connecting_paragraph_7",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": "../../linux-tutorial"
"0": "https://docs.hpc.ugent.be/linux-tutorial"
},
"previous_title": "account_paragraph_3",
"next_title": "account_paragraph_5",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": "../connecting"
"0": "https://docs.hpc.ugent.be/connecting"
},
"previous_title": "account_paragraph_6",
"next_title": "account_linux_paragraph_7.2",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "connecting",
"parent_title": "Transfer-Files-tofrom-the-HPC",
"links": {
"0": "https://docs.hpc.ugent.be/connecting/localhost:8000/Gent//intro-Linux/uploading_files/#symlinks-for-datascratch"
"0": "https://docs.hpc.ugent.be/localhost:8000/Gent//intro-Linux/uploading_files/#symlinks-for-datascratch"
},
"previous_title": "connecting_paragraph_12",
"next_title": "connecting_linux_paragraph_13.2",
Original file line number Diff line number Diff line change
@@ -6,8 +6,8 @@
"directory": "connecting",
"parent_title": "Transfer-Files-tofrom-the-HPC",
"links": {
"0": "",
"1": ""
"0": "https://docs.hpc.ugent.be/",
"1": "https://docs.hpc.ugent.be/"
},
"previous_title": "connecting_linux_paragraph_13.5",
"next_title": "connecting_linux_paragraph_13.7",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "connecting",
"parent_title": "First-Time-connection-to-the-HPC-infrastructure",
"links": {
"0": "https://docs.hpc.ugent.be/connecting/../troubleshooting/#warning-message-when-first-connecting-to-new-host"
"0": "https://docs.hpc.ugent.be/troubleshooting/#warning-message-when-first-connecting-to-new-host"
},
"previous_title": "connecting_paragraph_4",
"next_title": "connecting_linux_paragraph_5.2",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": "../../linux-tutorial"
"0": "https://docs.hpc.ugent.be/linux-tutorial"
},
"previous_title": "account_paragraph_3",
"next_title": "account_paragraph_5",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": "../connecting"
"0": "https://docs.hpc.ugent.be/connecting"
},
"previous_title": "account_paragraph_6",
"next_title": "account_macos_paragraph_7.2",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": "../../linux-tutorial"
"0": "https://docs.hpc.ugent.be/linux-tutorial"
},
"previous_title": "account_paragraph_3",
"next_title": "account_windows_paragraph_4.2",
Original file line number Diff line number Diff line change
@@ -6,8 +6,8 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": "https://docs.hpc.ugent.be/account/../connecting/#open-a-terminal",
"1": "https://docs.hpc.ugent.be/account/../account/#generating-a-publicprivate-key-pair"
"0": "https://docs.hpc.ugent.be/connecting/#open-a-terminal",
"1": "https://docs.hpc.ugent.be/account/#generating-a-publicprivate-key-pair"
},
"previous_title": "account_windows_paragraph_4.1",
"next_title": "account_windows_paragraph_4.3",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": ""
"0": "https://docs.hpc.ugent.be/"
},
"previous_title": "account_windows_paragraph_4.2",
"next_title": "account_windows_paragraph_4.4",
Original file line number Diff line number Diff line change
@@ -6,8 +6,8 @@
"directory": "account",
"parent_title": "Getting-ready-to-request-an-account",
"links": {
"0": "https://docs.hpc.ugent.be/account/../account/#generating-a-publicprivate-key-pair",
"1": "https://docs.hpc.ugent.be/account/../account/#generating-a-publicprivate-key-pair"
"0": "https://docs.hpc.ugent.be/account/#generating-a-publicprivate-key-pair",
"1": "https://docs.hpc.ugent.be/account/#generating-a-publicprivate-key-pair"
},
"previous_title": "account_windows_paragraph_6.1",
"next_title": "account_windows_paragraph_6.3",
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
"directory": "connecting",
"parent_title": "First-Time-connection-to-the-HPC-infrastructure",
"links": {
"0": "https://docs.hpc.ugent.be/connecting/../troubleshooting/#warning-message-when-first-connecting-to-new-host"
"0": "https://docs.hpc.ugent.be/troubleshooting/#warning-message-when-first-connecting-to-new-host"
},
"previous_title": "connecting_paragraph_3",
"next_title": "connecting_paragraph_5",
Loading