- The genral project ideas are online at: https://kaixu.me/projects/.
- Survey on the Analysis of User Interactions and Visualization Provenance
- Provide the background and overview of existing work.
- It has a website
- Visualise the online sensemaking provenance as vector sequences, i.e., apply provectories to browser history data;
- Visualise the Jupyter provenance as vector sequences, i.e., apply provectories to jupyter user log;
- You should also go through the readings for 'Project: Jupyter Extension'
- Improve the vector sequence visualisation so it becomes easier to see patterns and understand what users are doing.
- You should also go through the readings for 'project: browser extension'
- Capture and visualise the user interaction logs in Jupyter using a Jupyter extension (similar to verdant, not the provenance vectors)
- to support tasks in machine learning model building, such as hyperparameter tuning
- to support end users, such as artists, to use generative AI models, such as stable diffusion
- Create a browser extension to support the usage of LLM or generative AI models through their web interface
- Improve the design of the HistoryMap so the tree fits better within a Chrome side panel
- Create a browser extension that provide end-to-end support for online shopping
- Start with online research (can use HistoryMap)
- Automatically gather the required information, such as the price for different model
- with web scraping or chatGPT
- Create a visual comparison of different options
- A user study to compare the tree/hierarchy visualisation first used in SenesMap against the latest that is availalbe in popular browsers, such as tab grouping and chrome 'journey'.
- How to record the 'why': many rules/decisions are recorded without the 'why'. Overtime, the reasons become no longer valid, while the decisions/rules are still being followed, sometimes blindly. This project aims to see if there is any efficient way to record the 'why' that will not introduce too much extra work and can be easily used by other applications.
Below is a general structure, and please refer to the moodle page for your project module for specific format/requirements. If there is any difference, always follow the instructions from the module page.
It is not always a marked component, but very important to ensure a successful project. Think of it as a tool to hellp you think ahead and plan. Also, you will reuse most of it for the final report later.
- Introduction/background: the general problem area, such as using machine learning to support sensemaking
- The problem: the specific problem that you want to improve. For example, the exsting recommendations made by machine learning models are not very good.
- literature review (academic publications): what academics have done on this problem. More details in the literature review section below.
- related software, app, website, and other resources: some of the related efforts are not published in an academic paper. For example, not all the recommendaiton algorithms are discussed in a paper.
- The main goal here is to set your work in the context of existing works: what is the gap/missing piece.
- Your project idea:
- The main hypothesis, e.g., using large language models can improve the quality of recommendations
- The approach, e.g.,
- who are your targeted users? and what are their use case (i.e., what task will they use your tool for)?
- what large language model do you plan to use? why do you choose it?
- how do you obtain the model? do you have the hardware needed to run the model?
- will you need to retrain/update the model? do you have the labeled data for this? do you have hardware required for this?
- ...
- The evaluation method (following the same example):
- what is the definition of 'quality' and how do you measure it?
- do you plan to talk to/interview your users to understand their requirements?
- do you plan to do any user evaluations?
- ...
- Plan
- Agile and iterative:
- Gradually build up to the final prototype;
- Don't try to build the final prototype from the beginning
- Literature review (if there is more to do)
- User requirements:
- Talk to the real users
- Document their needs: task specific (e.g. finding the most suitable master degree), not technical specific (need to use machine learning)
- Paper prototype: design sketches
- on real paper or design software such as figma
- Get user feedback
- Working prototype with minimal functions (MVP: minimal viable product):
- leave out non-essentail functions (such as user account) and
- use simple alternatives instead of complex ones (e.g., using decisoin tree insteal of Large Language Models)
- Get user feedback
- Fully working prototype (final prototype)
- User evaluation
- Final report
- Agile and iterative:
- Create a project plan using GitHub Project (more on this later).
- Preliminary ethics form: to decide if you need to submit the full ethics form
- Full eithics form (most of you will need to): for this you will need
- recruitment materials
- information sheet
- consent form
- privacy notice
- Data Management Plan (DMP)
- More details: https://workspace.nottingham.ac.uk/pages/viewpage.action?spaceKey=CompSci&title=Research+Ethics+Guidelines+for+Academic+Staff%2C+Researchers+and+Students
We will this shared github reporsitory for project file and code, so you
- will never lose your code (because of corrupted USB or hard drive), and
- can easily show it to other people (e.g.,as part of your work portfolio for job application).
If you never use git before:
- concepts and commandline tool (git): https://git-scm.com/book/en/v2 (first 2 chapters)
- github (a website) tutorial: https://docs.github.com/en/get-started/quickstart/hello-world
You probably want to check out GitHub Student Developer Pack that provides features that you will need to pay otherwise.
How to use different GitHub features for the project:
- The repository is public, so don't upload any data that you don't want to share
- You can create a separate private repository for sensitve data. Follow the 'data management plan' of your ethics application.
- Please use this (wiki) page as the template for meeting notes
- Follow the meeting notes/minutes template there and the latest one on top.
- This guide can get you started if you never used markdown before
- It is similar to HTML, but much simpler. Also very useful for programing.
- Save the meeting note as a 'readme.md' (such as this file) under the appropriate folder (such as 2023-summer)
- 'Issues': todo list/tasks and coding issue
- Each 'issue' is a task to complete (action from the meeting)
- Assign an issue to the person that is responsible
- 'Projects': planning and project management (this has been updated recently, so points below may be updated)
- You can create a kanban board in 'project' (similar to Trello),
- create "milestones', e.g., one for each iteration
- add 'issues' to 'milestones', and set deadline
- start with this project template
- add your name to the project title, such as 'Kai's project plan'
- add your name to all the project issues, such as 'Kai - recruit users'
- Please make a subfolder under the appropriate folder, such as `2023-summer', for your project:
- Please send me your github account username or email, so I can add you to the repository.
- Please name your folder like this: year-degree-name-title (e.g., 2023-msc-kaixu-visual-analytics-for-sensemaking)
- Use the your folder for your code and other things like notes (in markdown)
- Include in your folder a 'readme.md' file to list all the things in your folder (e.g., which subfolder contains what) and other information such as how to run your code.
- Don't include large files as everyone else will need to download them as well (when they do a 'pull' of the repository).
- There will be top-level folder(s) for share resouces such as datasets.
Recommended: Visual Studio Code https://code.visualstudio.com/.
- This is suitable for both front end (JavaScript) and machine learing (Python) development;
- It is probably the most widely used IDE in industry;
- For front end, it is better for JavaScript development than other editors like Brackets (http://brackets.io/), Sublime Text (https://www.sublimetext.com/), or Notepad++ (https://notepad-plus-plus.org/)
- For machine learning, it can run python fine and can open Jupyter notebooks (need to install relevant plugins)
- Get started: https://code.visualstudio.com/docs
Recommended: Use Latex instead of Word for all the reports: proposal, progress report, final report, and viva.
- This will
- make your report look more professional
- make the references easier
- this is how many academic papers and books are written
- Recommended editor: Overleaf (https://www.overleaf.com/), like Google Docs for Latex.
- Learn Latex in 30 minutes: https://www.overleaf.com/learn/latex/Learn_LaTeX_in_30_minutes;
- The video recording of my overleaf tutorial: https://youtu.be/PqW8myobrPE
- How to add references: https://www.overleaf.com/learn/how-to/Using_bibliographies_on_Overleaf
- Needs to be much more complete than the litearture review in the project proposal;
- Cover different types of resources: app, website, service, organisation, physical print/product ...
- Focus on the most relevant: help you to narrow down your scope.
- For example, if your project is about AI-based recommendations, there is no need to introduce what AI is (the reader would know) and go straight to the latest machine learing models for recommendation.
- Include academic publications to support your argument:
- You can include webpages and online articles, but these are not 'peer reviewed', i.e., anyone can potentially say saything online, and as a result these are usually not regarded as reliable evidence.
- Please to search for academic papers:
- IEEE Xplore: http://ieeexplore.ieee.org/Xplore/home.jsp
- ACM Digital Library: https://dl.acm.org/
- Google Scholar: https://scholar.google.com/
- Things to consider when selecting which papers to cite
- IEEE Xplore and ACM Digital Library are usually more rigourous than Google Scholar: all papers published there are checked by one or more researchers, whereas this is not always the case for Google Scholar (such as self-publishing on arXiv).
- Papers that are more than 5 years old is often outdated (no longer the state of the art), especially in an active research field such as machine learning. Make sure there is no later work that supersedes it if you want to use it.
- Papers in well known journals and conferences, such as Nature or Science, are considered strong argument/evidence.
- You can also get an idea of how good a journal or conference is from its 'impact factor' (has different names) such as the 'h5' metrics used by Google or the 'impact score' by research.com.
For this project, you are likely to find good-quality related work from these journals and conference
- IEEE Transactions on Visualization and Computer Graphics (IEEE TVCG)
- Computer Graphics Forum
- ACM conference on human factors in computing systems (ACM CHI)
Besides search, it can be very useful to just go through the latest papers from these journals and conferences
- Because relevant papers do not necessarily include the same keywords (a paper about analysing user behaviours may not have the term 'sensemaking')
- Start from the latest issue/year, and then move onto the year before once completed, and so on.
- usually the last 3-5 years will be sufficient.
- Have a quick look of the title and/or abstract to check if the paper is relevant or not
- many papers can be excluded by the title alone
- save the paper in zotero if it seems relevant after reading the abstract
- We will use a shared zotero group to collect and share all the relevant papers found (more below in the 'reference' section)
- Be open about what form the project prototype will be: app, website, service, organisation, physical print/product ...
- Research different ways of implementation:
- For example for app development, there is native and hybrid, and then there are many different hybrid frameworks.
- Comparison:
- Identify required functions, such as user regiter/login, database, messaging, image/video, ...
- Compare different options regarding to the functions: is it possible at all, how easy it is to learn/program, how fast/secure/reliable it is, ...
- Justify your choice
- Two parts: citation in the main text and the references at the end of reprot.
- Use a tool; don't create reference manually.
- Recommended: zotero (see details below)
- Use zotero for collecting and inserting citations/references
- We will also use zotero to share relevant papers.
- Add all the relevant papers to this Zotero Group Library: Sensemaking and human-AI Teaming
- Use Zotero tags to group the papers (don't just use all the tags Zotero automatically created)
- Don't upload the full paper PDF file (the free zotero group library has a very limited storage size)
- Do include a link to the public/free online full paper pdf if there is one using the 'notes' section in Zotero.
- Use the 'notes' section to summarise what the paper is about and discuss its stregnth and weakneess.
- If you are using Latex,
- follow this overleaf bibliography tutorial;
- watch my latex tutorial, which includes references.
- you can use Zotero (see below) to collect the papers from publication webiste (such as IEEE xplore and ACM Digital Library) and then generates the bibtex entry.
- IEEE xplore and ACM Digital Library also provide the bibtex entry on the webpage for each paper: 'cite this' for Xplore, 'export citation' (double quote button) for ACM Digital library.
- If you are not using Latex, use Zotero.
- This is a recording of me showing how to use Zotero: https://youtu.be/R_5bjdE-kSw
- use their browser extension to automatically add reference:
- Zotero connector
- These can export the papers to bibtex format, if you use Latex.
- use their Microsoft Word plugin to add references to report
- Find real users or company/organisation/charity that work with real users.
- Find someone local if possible: much easier for the communication.
- Get the requirements from the real users, not what you think.
- The requirements should be technology independent, in most cases.
- 'How to Talk to Users' by YCombinator the most successful startup incubator (airbnb, dropbox, stripe, etc.) https://www.youtube.com/watch?v=MT4Ig2uqjTc
- Paper prototype/wireframe tool:
- Pencil and paper
- Low fidelity: Balsamiq https://balsamiq.com/
- High fidelity: Figma: https://www.figma.com
- Share the design: InVision https://www.invisionapp.com/
- Show user the paper prototype/wireframe and get feedback.
- Revise the desgin and project scope accordingly.
- Literature review should be more or less finished. This is a good time to discuss if there is any issue or difficulty.
- Everyone updates on:
- Any change on the project idea and/or scope and the reason;
- Any decision on development technology and the reason.
- Results on the mobile app development technology survey.
- Contacted any users, and what are their requirements?
- Any questions about the requirments, design, and evaluation?
- Please prepare a short demo of the progress so far.
- This shoud be informal, and no more than 5 minutes.
- live demo if prefererred, but screen recording and screen shot are also fine.
- This is a good practice for your viva demo
- You can edit this page or email me if there is anything you want to discuss before the meeting,
- Otherwise I will try to answer your questions then.
The meeting will have two parts:
Anything related to project, especially the evaluation, final report and viva.
- Recurit 10-15 people (the more the better);
- Collect some demographic information, such as age, gender, education level, and other things that are important to your app;
- Divide the participants into two groups: one using your app, one without;
- Ask them to complete one or more tasks (exact the same for two groups);
- Record some measurements: such as completion time, accuracy, and quality of the answer;
- Record the process: screen capture video or video recording;
- Compare the 'measurements' from the two groups to see if there is any difference;
- Analyse the recording to see what caused the difference and how it happened;
- The actual viva is usally 30 minutes, with 20 minutes presentation & demo and 10 minute Q&A
- The presentation (usually in ppt) covers everything about the project: from introduction and literature reveiw to implemantation and evaluation. Essentilly, it is an overview of your entire project in 15 minutes.
- There is ususally a demo if the project produces some software/app/website. This is your chance to impress the examiners (about the quality of the project outcome), as it provides much more details than the screenshots in the final report.
- The presentation and demo should be no more than 20 minutes in total, which means about 10 ppt slides. You can mix the two, i.e., including demo during presentation, or do one after the other, e.g., slides first and then demo.
- The last 10 minutes are for the examiner to ask questions. This can be to used to clarify things, ask for missing infomration, ensure you fully understand the work, etc.
For the mock viva,
- The project doesn't have to be completed; practice with what you have;
- Everyone can ask questions, not just Kai;
- Everyone will provide feedback on how to improve the presentatio and demonstration afterwards.
- Abstract (Mark out of 3)
- Background and Literature: problem definition, literature review etc (Mark out of 5)
- Design: requirements analysis and design specifications (Mark out of 6)
- Work done (Mark out of 8)
- Evaluation (Mark out of 8)
- Conclusion: with critical reflection on problem as defined (Mark out of 5)
- Presentation: Presentation of report and quality of written English (Mark out of 4)
- BCS criteria: Does the project satisfy the BCS exemption criteria (0 no, 1 yes)
- Quality of artefact: Is the artefact well designed, organised, and logically developed? Is the artefact sophisticated, polished, and elegant? Is the coding style advanced? How well does it meet its requirements in practice? Has it been evaluated sufficiently? (Mark out of 15)
- Presentation and understanding: Does the student understand all aspects of the work they have done (from requirements to evaluation?) Can they articulate and justify decisions they have made at all stages of the project. Can the student discuss other aspects of their work (e.g. background) (Mark out of 15)
- Abstract (Mark out of 3)
- Background and Literature: problem definition, literature review etc (Mark out of 5)
- Design: requirements analysis and design specifications (Mark out of 5)
- Work done (Mark out of 8)
- Evaluation (Mark out of 5)
- Conclusion: with critical reflection on problem as defined (Mark out of 5)
- Presentation: Presentation of report and quality of written English (Mark out of 4)
- BCS criteria: Does the project satisfy the BCS exemption criteria (0 no, 1 yes)
- Viva (Mark out of 25)
- Artefact Evaluation: Does the artefact appropriately culminate the work for the time period and study done? (Mark out of 20)