Lexos v3.2.0

v3.2.0 is the first version of Lexos in Python 3.6.x!

Code base changes:

• Refactored to a fully modularized code structure (Zhang et al., 2017).
  • Back-end: View, receiver, model structure.
  • Front-end: Flask blueprints.
• Python 3.6.x code integrated with type hinting.
• Used Pandas DataFrame as the Doc-Term Matrix data structure.
• Javascript uses newest ES6 features.
• All functions handled with AJAX calls.
• Updated jQuery to v3.x to resolve security problems.

Code quality checks:

• Python Code Checks:
  • Code style checked with Flake8 (PEP8/PEP257).
  • Unit test with pytest.
  • Doc style checked with PydocStyle.
• JS Code Checks:
  • Code style checked with ESLint with JavaScript Standard Style.
  • Doc style checked with ESLint JSDoc rules.

New functionalities, front-end redesigns and bug fixes.

• Added Bootstrap Consensus Tree (BCT) Beta.
• Added Content/Sentiment Analysis Beta.
• YouTube video instructions for most tools.
• Use bootstrap modal for all error messages to be consistent and error messages are clearer than before.
• Statistics: Added a new plotly box plot and the page layout is re-designed.
• K-Means: Switched to plotly for 2D-Scatter and Voronoi plots. Added a 3D-Scatter visualization method.
• Hierarchical Clustering: Switched to plotly library for visualizations.
• Rolling Window Analysis: Switched to plotly library for the graphing and added support for multiple milestones.
• Topword: Upgraded presentation of the existing document classes.
• Added a popover to highlight our In the Margins Scalar book.

Bug fixes

• In Tokenize section on analyze pages, fixed counts for temporary labels.
• Fixed spinning issue on Chrome.
• Fixed TF-IDF normalize options.
• Fixed download result button on Rolling Window Analysis.
• Corrected error message when an invalid file is uploaded on upload page.

Other changes

• Grey word functionality is removed.
• The "topic clouds" feature in the Multicloud tool, which can be used to analyze data from MALLET-produced topic models, has been temporarily removed. We hope to re-introduce it in the next release.

Citation information

Kleinman, S., LeBlanc, M.D., Drout, M., and Feng, W. (2018). Lexos. v3.2.0 https://github.com/WheatonCS/Lexos/

To see a summary of our refactoring work:
Zhang, C. ’18, Feng, W. ’19, Steffens, E. ’18, de Landaluce, A. ’17, Kleinman, S. and LeBlanc, M.D. (2018). Lexos 2017: Building Reliable Software in Python. The Journal of Computing Sciences in Colleges, v33(6), 124-134.

Lexos v3.1.1

Patches for

• Upload
• Multicloud
• ITM video display

This will be the last version in Python 2.7.

Summer 2016

• Moved to Bootstrap framework.
• Moved many functions to Ajax.
• Updated DataTables library.
• Improved error messages.
• More consistent labeling and small UI fixes.
• Added Twitter and Feedback buttons.
• New Local and Beta modes.
• Old Select tool replaced with new Manage tool.
• Scrubber: Added Remove whitespace, Keep Ampersands, Keepwords lists, and markup tag scrubbing functions (full support for XML, partial for HTML/SGML).
• Tokenizer: New version uses serverside processing to handle large DTMs efficiently. New Show/Hide columns function.
• WordCloud, Multicloud, Bubbleviz: Improved tooltips.
• WordCloud: Term counts table switched to DataTables with search and sort functions.
• Multicloud: New toggle switch for topic clouds.
• Rolling Windows: Reorganized UI display.
• Statistics: Switched to DataTables with client-side download buttons for multiple formats.
• Hierarchical Clustering: New Silhouette Score interface, dendrogram is now a PNG by default, new Newick format export button.
• K-Means Clustering: Interactive tooltips for Voronoi cells for better readability.
• Similarity Query: More efficient code. Output now uses DataTables.
• TopWords: Enhanced functionality and clearer interface.

Citation information:
Kleinman, S., LeBlanc, M.D., Drout, M., and Zhang, C. (2016). Lexos v3.0. https://github.com/WheatonCS/Lexos/.

Auto-Installers:

• Windows: lexos_installer.exe
• Mac OS: Mac_Installer_Bundle.tar.gz

Manual Install Directions:

• Windows: WindowsInstallGuide.pdf
• Mac OS: MacOSXInstallGuide.pdf

Summer 2015

New paint for the UI look and feel.
Reworked Rolling Window Analysis
Kmeans Clustering got some love
Document Milestones if inserted beforehand can be detected on cut
Tokenize moved to DataTables
Top-Words added

Citation information:
Kleinman, S., LeBlanc, M.D., Zhang, C. (2015). Lexos. v. 2.5. https://github.com/WheatonCS/Lexos/.

Summer2014

Following a large overhaul of the back-end, this is our 2.0.0 release. We've changed a lot of how the python works on the server side to allow an easier process for adding new analysis tools, as well as allowing ourselves to offer more functionality to the existing ones. This release doesn't contain a whole lot in terms of building on top of that new back-end, but it does include some new features, including:

• Re-structuring the site to emphasize the stages of text analysis, mostly present in the nav-bar.
• New section of the site dedicated to "Management". Contains entirely pre-existing tools.
• More careful attention to non-western languages when uploading and tokenizing.
• New section of the site dedicated to "Preparation". Contains entirely pre-existing tools.
• New section of the site dedicated to "Visualizations". Contains updated pre-existing tools and a new Rolling window tool.
• New tokenization options: word n-grams and character n-grams; tf-idf weighting.
• New section of the site dedicated to "Analysis". Contains pre-existing tools and addition of k-means clustering, silhouette scores on clusters, and similarity query tool.

There have been minor changes throughout the site, as well as a layout shift, in order to better use screen space on various screen/browser sizes. Please report any and all issues on our repo.

Citation information:
LeBlanc, M.D., Jensen, B., Kleinman, S. (2014). Lexos. v. 2.0. https://github.com/WheatonCS/Lexos/.

Lexos Release 2.0

Summer2013

Lexos

This is the first real release version of Lexos, which has been live on our server for quite a few months. This version comprises all of the work from the summer of 2013 and predates much of our attempts at redesigning and overhauling the backend to allow for more scalability and more flexibility in our offered features. It depends on the FlatUI framework, which in turn relies on Bootstrap, and has a very polished and professional look, one that is unlikely to change in future releases.

Features

• Upload
  • Allows the user to put their files into the Lexos tool for processing and analysis.
• Manage
  • Allows the user the enable/disable files for use in the tool. Disabled files will not be affected by any actions taken on the site.
• Scrub
  • Allows the user to perform normalizing actions on their text such as converting all characters to lowercase, removing punctuation, etc.
• Cut
  • Allows the user to cut their files into chunks, useful for analyzing against each other using the dendrogram analysis option.
• Analyze
  • CSV-Generation
    • Allows the user to generate a csv from their files, useful for exporting the results from our tool to other tools that aren't compatible with ours.
  • Dendrogram
    • An unsupervised machine learning technique for analyzing the similarity of different files. Useful in conjunction with cutting files.
  • WordCloud
    • Not technically an analysis tool, this visualization can be useful for identifying key aspects of texts through visual representation of the frequency of the words contained therein.
  • BubbleViz
    • Similar to WordCloud, but instead of scaling the words themselves, provides varying-sized circular "bubbles" to depict word frequency.
  • MultiCloud
    • A scalable version of WordCloud, this tool can take multiple files and create word clouds from each. Useful for determining the key aspects/differences between two files by identifying them side-by-side.

We also include links to older tools, developed by Wheaton students and in past years of the work done in the (Lexomics Research Group)[http://wheatoncollege.edu/lexomics/], which aren't currently integrated into the Lexos platform/workflow.

Bugs

Please report any and all bugs on our GitHub repo, under the issues section.