Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parser script for the chatbot input #675

Open
wants to merge 161 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
161 commits
Select commit Hold shift + click to select a range
1ebc363
initial commit
EwDa291 Aug 8, 2024
34df842
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 8, 2024
10edb20
some cleanup
EwDa291 Aug 8, 2024
85a93ec
used jinja to replace macros
EwDa291 Aug 9, 2024
dfff5fa
adapt if-mangler to accommodate for nested if-clauses
EwDa291 Aug 9, 2024
649ddec
adapt the parser to take all files as input, not all files get parsed…
EwDa291 Aug 9, 2024
2116d6e
adapt the parser to take all files as input, not all files get parsed…
EwDa291 Aug 9, 2024
159aa62
small update, not important
EwDa291 Aug 9, 2024
75765e5
change to the templates
EwDa291 Aug 9, 2024
57d9cfe
change to accommodate for more nested if-clauses
EwDa291 Aug 9, 2024
75d345b
Delete scripts/HPC chatbot preprocessor/start_checker.py
EwDa291 Aug 9, 2024
ff7a9fc
make sure files with duplicate names between normal files and linux-t…
EwDa291 Aug 12, 2024
47a33b7
Merge branch 'chatbot_parser' of https://github.com/EwDa291/vsc_user_…
EwDa291 Aug 12, 2024
7d279d6
fixed the problem of some files being written in reST instead of mark…
EwDa291 Aug 12, 2024
8047572
some small fixes
EwDa291 Aug 12, 2024
7d1c5ed
remove try-except-structure
EwDa291 Aug 13, 2024
984b0cd
collapse all code into one file
EwDa291 Aug 13, 2024
8f5eeaa
Rename file
EwDa291 Aug 13, 2024
2b97b7a
cleanup repository
EwDa291 Aug 13, 2024
b595301
Rename directory
EwDa291 Aug 13, 2024
90c8ab7
add a main function
EwDa291 Aug 13, 2024
b8ae706
make file paths non os-specific
EwDa291 Aug 13, 2024
b751497
use docstrings to document the functions
EwDa291 Aug 13, 2024
0f8eb5d
rewrite the if-mangler to make it more readable
EwDa291 Aug 13, 2024
9938e92
got rid of most global variables
EwDa291 Aug 13, 2024
508b22c
fixed some issues with if statements
EwDa291 Aug 13, 2024
a25ce2d
fixed some issues with if statements
EwDa291 Aug 13, 2024
80d0535
got rid of all global variables
EwDa291 Aug 13, 2024
9163a75
small changes to make file more readable
EwDa291 Aug 14, 2024
1dcffc1
codeblocks, tips, warnings and info reformatted
EwDa291 Aug 14, 2024
4d7fbdb
small optimisations
EwDa291 Aug 14, 2024
671f7f3
small optimisations
EwDa291 Aug 14, 2024
e5c39bd
initial commit
EwDa291 Aug 14, 2024
c6492fc
added requirements
EwDa291 Aug 14, 2024
aff8198
added requirements and usage info
EwDa291 Aug 14, 2024
a981002
minor changes to the print statements
EwDa291 Aug 14, 2024
1f3b343
reworked function to take care of html structures
EwDa291 Aug 16, 2024
b6388d3
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 16, 2024
48cad97
filter out images
EwDa291 Aug 16, 2024
df58f23
get rid of backquotes, asterisks, pluses and underscores used for for…
EwDa291 Aug 16, 2024
c423e07
dump to json files instead of txt files
EwDa291 Aug 16, 2024
2c333fe
cleaned up parser with macros
EwDa291 Aug 16, 2024
ce52352
cleaned up parser with macros
EwDa291 Aug 16, 2024
5db34af
cleaned up parser with macros
EwDa291 Aug 16, 2024
4226d28
Update README.md
EwDa291 Aug 19, 2024
d730a26
Update README.md
EwDa291 Aug 19, 2024
f3182e3
added section about restrictions on input files
EwDa291 Aug 19, 2024
aee54de
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 19, 2024
675bec5
adapted section about restrictions on input files
EwDa291 Aug 19, 2024
f1e58ef
adapted section about restrictions on input files
EwDa291 Aug 19, 2024
2bf1075
Merge branch 'chatbot_parser' of https://github.com/EwDa291/vsc_user_…
EwDa291 Aug 19, 2024
a168509
change variables to be lowercase
EwDa291 Aug 19, 2024
09b86c9
take out some copy pasting
EwDa291 Aug 19, 2024
f95b99e
added warning about long filepaths
EwDa291 Aug 19, 2024
06bb7b9
fixing typos
EwDa291 Aug 19, 2024
2f3e5b3
take out copy pasting
EwDa291 Aug 19, 2024
0c4dbe8
first draft version of the restructured script to accommodate for the…
EwDa291 Aug 20, 2024
38c4572
added support to filter out collapsable admonitions
EwDa291 Aug 20, 2024
5cbd653
attempt at fix for problems with jinja include, not working yet
EwDa291 Aug 20, 2024
0e6f8b2
fixed an issue with jinja templates
EwDa291 Aug 21, 2024
cd77837
added docstrings to new functions
EwDa291 Aug 21, 2024
98eb695
only add necessary if-statements in front of non-if-complete sections
EwDa291 Aug 21, 2024
27457e3
fixed some more jinja problems
EwDa291 Aug 21, 2024
bb72287
implemented extra test to make sure generic files dont accidentally g…
EwDa291 Aug 21, 2024
67cb19e
make sure empty os-specific files are not saved
EwDa291 Aug 21, 2024
cf9834a
clean up unused code
EwDa291 Aug 21, 2024
da32459
introduce more macros
EwDa291 Aug 21, 2024
093200b
reintroduce logic to remove unnecessary directories
EwDa291 Aug 21, 2024
5d0ffe9
added functionality to include links or leave them out
EwDa291 Aug 21, 2024
a3e34a9
added functionality to include links or leave them out
EwDa291 Aug 21, 2024
7c6154b
adapt filenames to allow for splitting on something other than subtitles
EwDa291 Aug 21, 2024
8d5b50d
making some changes to prepare to add paragraph level splitting tomorrow
EwDa291 Aug 21, 2024
0c10376
making some changes to prepare to add paragraph level splitting tomorrow
EwDa291 Aug 21, 2024
f8ee860
making some changes to prepare to add paragraph level splitting tomorrow
EwDa291 Aug 21, 2024
6533733
adapted the parsing script to allow for testing in a semi-efficient way
EwDa291 Aug 21, 2024
2e7a00f
added test for make_valid_title
EwDa291 Aug 21, 2024
f5e0579
removed useless lines from testscript
EwDa291 Aug 21, 2024
6757b4f
First attempt at splitting in paragraphs (need for other fixes for ti…
EwDa291 Aug 22, 2024
6d9558d
make two functions for different ways of dividing the text
EwDa291 Aug 22, 2024
2c7025a
added docstrings to new functions
EwDa291 Aug 22, 2024
ae99bb9
update test for valid titles
EwDa291 Aug 22, 2024
084b421
fixed problem with splitting os-specific text (metadata not fixed yet)
EwDa291 Aug 22, 2024
cf7f5f0
fix for metadata of os-specific sections
EwDa291 Aug 22, 2024
b7c10d3
clean up temporary version
EwDa291 Aug 22, 2024
4a441f3
added command line options for custom macros
EwDa291 Aug 22, 2024
662134f
small fix to macros
EwDa291 Aug 22, 2024
05eab4a
clean up test for valid title
EwDa291 Aug 22, 2024
b85a8fb
add a test for write_metadata
EwDa291 Aug 22, 2024
39a3c99
added functionality to split on paragraphs
EwDa291 Aug 23, 2024
af9e6cc
clean up
EwDa291 Aug 23, 2024
f4163a7
clean up
EwDa291 Aug 23, 2024
833f964
further clean up and added shebang
EwDa291 Aug 23, 2024
79b1a56
clean up
EwDa291 Aug 23, 2024
cec154c
added test for if mangler
EwDa291 Aug 23, 2024
2f4a277
clean up
EwDa291 Aug 23, 2024
cd0c8eb
clean up customizable options
EwDa291 Aug 23, 2024
3be262a
further adapt the script to be able to test it
EwDa291 Aug 26, 2024
1d32aab
make changes to usage in command line to be more intuitive
EwDa291 Aug 26, 2024
5902c96
first revised version of the README
EwDa291 Aug 26, 2024
6f97d5f
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 26, 2024
6e48800
added docstring to main function
EwDa291 Aug 26, 2024
0bc440b
include chatbot_prepprocessor
EwDa291 Aug 26, 2024
e6e6023
added options for source and destination directories
EwDa291 Aug 26, 2024
a6d99d9
cleanup
EwDa291 Aug 26, 2024
2be834f
cleanup
EwDa291 Aug 26, 2024
532543a
cleanup
EwDa291 Aug 26, 2024
107464e
relocate test files
EwDa291 Aug 26, 2024
dd64381
update arguments of if mangler
EwDa291 Aug 26, 2024
ef3fd58
relocate full test files
EwDa291 Aug 26, 2024
4d7db8f
Revert "update arguments of if mangler"
EwDa291 Aug 26, 2024
df9bac5
Revert "relocate full test files"
EwDa291 Aug 26, 2024
631d9e9
update test to adapt to new arguments in if mangler
EwDa291 Aug 26, 2024
c6e600d
relocated full test files
EwDa291 Aug 26, 2024
d1c6194
Rename test_paragraph_split_1.md to test_paragraph_split_1_input.md
EwDa291 Aug 26, 2024
695ffd6
Rename test_title_split_1.md to test_title_split_1_input.md
EwDa291 Aug 26, 2024
af4832b
smal fix
EwDa291 Aug 26, 2024
8805c8c
test text for paragraph split
EwDa291 Aug 26, 2024
a265ffd
start of a fix for double title problem, not done yet
EwDa291 Aug 26, 2024
6c2a61c
Fix for double title bug when splitting on paragraph
EwDa291 Aug 27, 2024
ed08879
Fix bug for empty linklist in metadata
EwDa291 Aug 27, 2024
176af13
fix bug where too many directories were sometimes created
EwDa291 Aug 27, 2024
d4ceac8
test of full script, test files not ready to be pushed yet
EwDa291 Aug 27, 2024
815a863
updated requirements.txt
EwDa291 Aug 27, 2024
d15469f
updated docstring in main function
EwDa291 Aug 27, 2024
daa6b36
add support for comments for the bot to be included in the source files
EwDa291 Aug 27, 2024
4c19f44
changed the default for min paragraph length
EwDa291 Aug 27, 2024
9a6ff58
added test files for full script test
EwDa291 Aug 27, 2024
56543f0
small fix for double title bug
EwDa291 Aug 27, 2024
52a3861
added examples of output of the script when splitting on paragraphs w…
EwDa291 Aug 27, 2024
692e77b
fix for issue with html links
EwDa291 Aug 27, 2024
7f493a1
fix for issue with html links
EwDa291 Aug 27, 2024
0e34396
fix for issue with relative links to the same document
EwDa291 Aug 27, 2024
fa00044
added test for replace_markdown_markers
EwDa291 Aug 27, 2024
b3952b2
fix to small inconsistency in metadata
EwDa291 Aug 27, 2024
73072bf
added test for insert_links
EwDa291 Aug 27, 2024
3161309
make sure paragraphs only include full lists
EwDa291 Aug 28, 2024
7d4d7f9
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 28, 2024
3407be3
adapted to the new source files
EwDa291 Aug 28, 2024
6d04bbc
add source-directory to metadata and verbose mode
EwDa291 Aug 28, 2024
f33cfb3
added verbose mode
EwDa291 Aug 28, 2024
1c389d7
Merge branch 'hpcugent:main' into chatbot_parser
EwDa291 Aug 28, 2024
3227f19
Added limitation on lists
EwDa291 Aug 29, 2024
67aed53
fix for non os-specific if-statement not being recognised
EwDa291 Aug 29, 2024
9e297b1
new test for links
EwDa291 Aug 29, 2024
b6b8610
new test to make sure lists are kept as one section
EwDa291 Aug 29, 2024
57a2139
updated test_file for list test
EwDa291 Aug 29, 2024
170a10c
dropped <> around links and started new function to calculate length …
EwDa291 Aug 30, 2024
04efff6
removed parsed mds
EwDa291 Aug 30, 2024
1ef1f10
Changed paragraphs to decide length based on tokens instead of charac…
EwDa291 Aug 30, 2024
621c0a3
Changed paragraphs to decide length based on tokens instead of charac…
EwDa291 Aug 30, 2024
adf364d
Changed paragraphs to decide length based on tokens instead of charac…
EwDa291 Aug 30, 2024
2c6e3c4
Merge branch 'hpcugent:main' into chatbot_parser_code
EwDa291 Aug 30, 2024
32b8b74
removing unnecessary files
EwDa291 Aug 30, 2024
445f7ee
Revert "removing unnecessary files"
EwDa291 Aug 30, 2024
d96aeba
change filenames
EwDa291 Aug 30, 2024
bec9a63
Updated all tests to incorporate the latest changes
EwDa291 Aug 30, 2024
886da19
change working directory of test lists
EwDa291 Aug 30, 2024
b354bb2
change directory in metadata to linux version
EwDa291 Aug 30, 2024
87e5575
change directory in metadata to linux version
EwDa291 Aug 30, 2024
8f39e08
add pytest to requirements
EwDa291 Aug 30, 2024
d6c33e7
change test for write_metadata
EwDa291 Aug 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixed some issues with if statements
  • Loading branch information
EwDa291 committed Aug 13, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 508b22c7b3a485f8fdb64059a45d2ee9dfdd4f04
31 changes: 21 additions & 10 deletions scripts/HPC_chatbot_preprocessor/chatbot_parser.py
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@
################### define global variables ###################

# variable that keeps track of the source directories
source_directories = [os.path.join("..", "..", "mkdocs", "docs", "HPC"), os.path.join("..", "..", "mkdocs", "docs", "HPC", "linux-tutorial")]
source_directories = [os.path.join("..", "..", "mkdocs", "docs", "HPC")]#, os.path.join("..", "..", "mkdocs", "docs", "HPC", "linux-tutorial")]

# list of all the filenames
filenames_generic = {}
@@ -189,7 +189,7 @@ def mangle_os_ifs(line, is_os):

constr_match = re.search(r'\{%.*?%}', match.string)
if_match = re.search(r'if ', match.group(1))
if_os_match = re.search(r'if OS == ', match.group(1))
if_os_match = re.search(r'if OS ', match.group(1))
endif_match = re.search(r'endif', match.group(1))
pos_first_mangle = constr_match.start() + start_index + added_length + 1
pos_second_mangle = constr_match.end() + start_index + added_length - 1
@@ -228,18 +228,18 @@ def mangle_os_ifs(line, is_os):
return line, is_os


def mangle_ifs(directory, file):
def mangle_ifs(directory, filename):
"""
function that writes the if-mangled version of a file to a location where the jinja parser will use it

:param directory: the directory of the file to be if mangled
:param file: the filename of the file to be mangled
:param filename: the filename of the file to be mangled
:return:
"""
# variable to keep track of latest if-statement scope
is_os = 0 # Can be 0, 1, 2 or 3 {0: not in an os-if; 1: in a non-os-if nested in an os-if; 2: in an os-if; 3: in an os-if nested in an os-if}

with open(os.path.join("if_mangled_files", file), 'w') as write_file:
with open(os.path.join("if_mangled_files", filename), 'w') as write_file:
with open(directory, 'r') as read_file:
for line in read_file:
new_line, is_os = mangle_os_ifs(line, is_os)
@@ -261,17 +261,17 @@ def check_if_statements(curr_line, active_OS_if_states):
:return: the text to be written to the file, if any
"""
# check whether the first part of the line contains information wrt if-statements
match = re.search(r'^\{-if-%([^%]*)%-if-}(.*)', curr_line)
match = re.search(r'^\{-if-%(.*?)%-if-}(.*)', curr_line)

# check whether the line contains information wrt if-statements that is not in its first part
match_large = re.search(r'^(.*)(\{-if-%[^%]*%-if-})(.*)', curr_line)
match_large = re.search(r'^(.*)(\{-if-%.*?%-if-})(.*)', curr_line)

if match:
content = match.group(1)

# new if-statement wrt OS
if re.search(r'if OS == ', content):
OS = content[9:-1]
OS = content.split()[-1]

# set new active OS
active_OS_if_states[OS] = "active"
@@ -281,6 +281,17 @@ def check_if_statements(curr_line, active_OS_if_states):
if other_OS != OS and active_OS_if_states[other_OS] == "active":
active_OS_if_states[other_OS] = "inactive"

elif re.search(r'if OS != ', content):
OS = content.split()[-1]

# set new active OS
active_OS_if_states[OS] = "inactive"

# set other inactive ones on active
for other_OS in active_OS_if_states.keys():
if other_OS != OS and active_OS_if_states[other_OS] == "inactive":
active_OS_if_states[other_OS] = "active"

# endif statement wrt OS
elif re.search(r'endif', content):
if str(1) in active_OS_if_states.values():
@@ -556,8 +567,8 @@ def main():
write_end_of_file(os.path.join(root_dir_os_specific_macos, last_directory, last_title + ".txt"), "macOS",
links_macos, is_linux_tutorial, main_title, last_title)

remove_directory_tree("copies")
remove_directory_tree("if_mangled_files")
# remove_directory_tree("copies")
# remove_directory_tree("if_mangled_files")


main()
Loading