From 4423b9d813f4dbc6ceba7104fd3f566a05c1602c Mon Sep 17 00:00:00 2001 From: Joe Carstairs Date: Tue, 28 Nov 2023 10:46:43 +0000 Subject: [PATCH] Example change --- .../2010-12-22-sorted_lists_in_java-copy.html | 293 ++++++++++++++++++ ...11-09-7-reasons-i-love-open-source-copy.md | 30 ++ ...2019-04-18-cloud-as-a-value-driver-copy.md | 31 ++ _posts/2023-02-07-state-of-open-con-copy.md | 24 ++ ...-of-encryption-as-we-know-it-copy.markdown | 44 +++ ...23-09-19-metrics-collector-in-jest-copy.md | 222 +++++++++++++ ...he-state-of-webassembly-2023-copy.markdown | 166 ++++++++++ 7 files changed, 810 insertions(+) create mode 100644 _posts/2010-12-22-sorted_lists_in_java-copy.html create mode 100644 _posts/2018-11-09-7-reasons-i-love-open-source-copy.md create mode 100644 _posts/2019-04-18-cloud-as-a-value-driver-copy.md create mode 100644 _posts/2023-02-07-state-of-open-con-copy.md create mode 100644 _posts/2023-04-03-beyond-the-hype-y2q-the-end-of-encryption-as-we-know-it-copy.markdown create mode 100644 _posts/2023-09-19-metrics-collector-in-jest-copy.md create mode 100644 _posts/2023-10-18-the-state-of-webassembly-2023-copy.markdown diff --git a/_posts/2010-12-22-sorted_lists_in_java-copy.html b/_posts/2010-12-22-sorted_lists_in_java-copy.html new file mode 100644 index 0000000000..7f733e18e9 --- /dev/null +++ b/_posts/2010-12-22-sorted_lists_in_java-copy.html @@ -0,0 +1,293 @@ +--- +title: Sorted Lists in Java +date: 2010-12-22 00:00:00 Z +categories: +- mrhodes +- Tech +tags: +- Algorithms +- Data Structures +- Java +- blog +id: 76695 +author: mrhodes +oldlink: http://www.scottlogic.co.uk/2010/12/sorted_lists_in_java/ +layout: default_post +source: site +disqus-id: "/2010/12/sorted_lists_in_java/" +summary: This post goes through an implementation a SortedList in Java which ensures + its elements are kept in order and guarantees O(log(n)) run time for all the basic + operations. +--- + +

This post goes through an implementation a SortedList in Java which ensures its elements are kept in order and guarantees O(log(n)) run time for all the basic operations: get, contains, remove and add. The source code, javadoc, unit tests and class files for this post are available here: scottlogic-utils-mr-1.4.zip.

+

Sorting is one of the most common operations applied to lists and as such Java has built in mechanisms for doing it, like the Comparable and Comparator interfaces and the Collections.sort methods. These are great when you have a static list that needs to be ordered, but sometimes you want the list to remain sorted after some altering operations have been applied to it (e.g. if you add an element to an ArrayList which you've sorted, chances are that it's no longer in the right order). For some reason, the java.util package is lacking a proper SortedList, and since they're quite handy, I thought I write my own.

+

AlternativesAs with all data structures, whether a SortedList the right tool for the job depends on how the data is going to be used. The java.util package contains a host of different data structures, all of which have their place, but unfortunately it (at least at the moment) is missing the SortedList. A comparison between some of the the Java's built in data structures and a SortedList is given below:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
add(Object elem)remove(Object elem)get(int index)contains(Object elem)multiple equal elements
ArrayListO(1)*O(n)O(1)O(n)YES
LinkedListO(1)O(n)O(n)O(n)YES
TreeSetO(log(n))O(log(n))N/AO(log(n))NO
PriorityQueueO(log(n))O(n)N/AO(n)YES
SortedListO(log(n))O(log(n))O(log(n))O(log(n))YES
* - amortized constant time (inserting n objects takes O(n) time).
+

If you're not likely to change the data structure much, you might want to just use an ArrayList or a regular array and sort it when you need, which can be done relatively quickly (O(n*log(n))), using Java's Collections.sort or Arrays.sort methods respectively. So long as you don't need to sort it too often, there's no problem.

+

If you're not sure how many times you'll need to sort it, it's quicker to ensure that the data is always kept in order. If you don't need to store multiple equal elements and don't need random access to the data (i.e. you don't need to run get(int index)) then it's probably best to use Java's TreeSet. If you need to store multiple equal elements but don't need random access to the data, or quick contains/remove methods then Java's PriorityQueue might be the way to go. If these cases don't apply, then a SortedList might be what you want.

+

Implementing a custom List in JavaAll Lists in Java should implement the java.util.List interface, however, rather than implement this directly, the easiest way to start off writing your own List class is have it extend the AbstractList class instead. This class implements the List interface and provides default implementations of the methods, reducing the amount of code you need to write. Most of these default implementations just throws an UnsupportedOperationException, but some are useful. For example, the default listIterator and iterator methods provide you with working iterators once you've provided an implementation for the get(int index) method.

+

It's also easy to configure the iterators provided by the AbstractList to exhibit fail-fast behaviour. This means that if the list is modified after the iterator has been created, other than through the iterator itself, then calling any method other than hasNext() on the iterator will (hopefully!) cause the cause a ConcurrentModificationException to be thrown, as is standard for all of Java's non-synchronized Collection classes. To do this, you just need to increment the modCount variable whenever you add or remove an element from the list. For example, the add method in the given implementation has the following structure:

+{% highlight java %} +public boolean add(T object){ + boolean treeAltered; + if(object != null){ + //add the object to the list.. + ... + treeAltered = true; + modCount++; + } + return treeAltered; +} +{% endhighlight %} +

Here I took the decision not to allow null values to be added in the list, just to simplify the other methods and since on the vast majority of applications this is what you want. If you really need to add a null value you can always just add a special singleton instance of type T which you know represents null instead of null itself. The effect of the calling modCount++; in the add method is that the following code will now throw a ConcurrentModificationException.

+{% highlight java %} +SortedList list = new SortedList(); +Iterator itr = list.iterator(); +list.add(1); +itr.next(); +{% endhighlight %} +

Another thing to consider when writing a custom List class (as with any class!) is how you are going to define its type and the constructors. Since a SortedList needs a way of comparing the elements it stores, I've decided to leave the type definition simple but only supply a constructor which takes a Comparator, this constructor has the following signature:

+{% highlight java %} +public SortedList(Comparator comparator) +{% endhighlight %} +

If you're not used to Java generics then this line might look a bit odd! The type definition, "<? super T>>" just says that the given comparator must be capable of comparing elements of type T, which is the type of element that the list is able to store.

+

This might seem like a weird decision, since Java's built in TreeSet doesn't enforce the same restriction; it also allows a no-argument constructor. The reason is that this no-argument constructor is used to create a TreeSet which orders elements by their natural order; i.e. the order you get if you use the compareTo method on the set's elements. The draw back of this design decision is that there is no way to say that either the elements must be Comparable, or you need to supply a Comparator, so you need to add a runtime check for this (i.e. note the ClassCastException thrown by the TreeSet's add method).

+

If you need this behaviour with this SortedList, you can you implement a very simple subclass of it which passes in a Comparator providing this natural ordering. An example implementation of this is shown below:

+{% highlight java %} +public class SortedListNatural> extends SortedList { + public SortedListNatural(){ + super(new Comparator(){ + public int compare(T one, T two){ + return one.compareTo(two); + } + }); + } +} +{% endhighlight %} +

Note that the type definition restricts the class so that only comparable objects can be stored in it; removing the need for any runtime check.

+

AVL Trees as Sorted ListsIn order to obtain the logarithmic run times for the standard operations, the SortedList needs to be based on some kind of balanced tree structure. I've used an AVL tree, which is pretty much the simplest form of balanced tree you can have (so less chance of mistakes!) and since it ensures the tree remains very balanced (more so than say a Red-Black Tree), it means that that the get and contains methods always run efficiently.

+

An AVL tree is a binary search tree, which rebalances itself whenever the height of any node's subtree becomes at least two larger than it's other subtree. The rebalancing requires implementing just two methods - the left and the right tree rotations. I won't go into all the details here, but if you're interested, a pretty easy to follow introduction to AVL trees is available at: lecture1, lecture2 and lecture3.

+

When it comes to actually implementing an AVL tree, the most obvious way to do it in Java is to have an inner Node class to represent the individual positions in the tree, and then have the main class hold a reference to the root Node of the tree. The Node class needs to be defined somewhat recursively, as each Node needs to maintain references to both the children nodes and their parent node. In order to use an AVL tree as a SortedList, you need this Node class to be slightly different than in a standard implementation, the changes required are that:

+
    +
  1. Allow more than one node to store values that are equal in terms of the given comparator.
    2. +
  2. Nodes need to remember the total number of children they have
    3. +
+

The first alteration is required so that the tree can be used as a List rather than as a Set and the second in order to implement the get(int index) method efficiently.

+

The start of the Node class in the implementation looks like this:

+{% highlight java %} +private int NEXT_NODE_ID = Integer.MIN_VALUE; + +private class Node implements Comparable { + final int id = NEXT_NODE_ID++; //get the id and increment it.. + T value; //the data value being stored at this node + + Node leftChild; + Node rightChild; + Node parent; + + //The "cached" values used to speed up methods.. + int height; + int numChildren; + + //Compares the t values using the comparator, if they are equal it uses the + //node it - older nodes considered to be smaller.. + public int compareTo(Node other){ + int comparison = comparator.compare(value, other.value); + return (comparison == 0) ? (id-other.id) : comparison; + } + ... +{% endhighlight %} +

As the Node class is an inner class there is no need to "parameterise" the type definition, it automatically inherits the definition of the T from the SortedList parent class. The list allows multiple values to be stored by giving each node a unique id, which is incremented as each element is added. The Node's compareTo method then uses this when comparing values, so that nodes with the same value according to the comparator, are distinguished by their unique id. The height and numChildren fields are really just cached values, since their values could be obtained by examining the child nodes. It's up to the implementation to ensure that these values are maintained as changes are made to the tree. In the given implementation, this is all done in the updateCachedValues method of the Node class:

+{% highlight java %} +private void updateCachedValues(){ + Node current = this; + while(current != null){ + if(current.isLeaf()){ + current.height = 0; + current.numChildren = 0; + } else { + //deal with the height.. + int leftTreeHeight = (current.leftChild == null) ? 0 : current.leftChild.height; + int rightTreeHeight = (current.rightChild == null) ? 0 : current.rightChild.height; + current.height = 1 + Math.max(leftTreeHeight, rightTreeHeight); + + //deal with the number of children.. + int leftTreeSize = (current.leftChild == null) ? 0 : current.leftChild.sizeOfSubTree(); + int rightTreeSize = (current.rightChild == null) ? 0 : current.rightChild.sizeOfSubTree(); + current.numChildren = leftTreeSize + rightTreeSize; + } + //propagate up the tree.. + current = current.parent; + } +} +{% endhighlight %} +

So long as this method is called on the appropriate node each time the tree is structurally altered, the values will remain correct. It's not always obvious which node constitutes the "appropriate one", but it should always be the node which was altered with the lowest height in the resulting tree (it's always the case that there is one such node).

+

The only key method that is missing from a standard AVL tree implementation that is required to make it work as a List is the get(int index) method. As I mentioned before, this method is going to make use of the numChildren field of the Node class to so that it can be implemented efficiently. Once this field is in place, it's not difficult to implement - the method just needs to traverse the tree, making sure that it remembers how many smaller values there are than those at the current node; this effectively tells you the index of the first value stored at the current node. In the provided implementation, the code look like this:

+{% highlight java %} +@Override +public T get(int index){ + return findNodeAtIndex(index).value; +} + +private Node findNodeAtIndex(int index){ + if(index < 0 || index >= size()){ + throw new IllegalArgumentException(index + " is not valid index."); + } + Node current = root; + //the the number of smaller elements of the current node as we traverse the tree.. + int totalSmallerElements = (current.leftChild == null) ? 0 : current.leftChild.sizeOfSubTree(); + while(current!= null){ //should always break, due to constraint above.. + if(totalSmallerElements == index){ + break; + } + if(totalSmallerElements > index){ //go left.. + current = current.leftChild; + totalSmallerElements--; + totalSmallerElements -= (current.rightChild == null) ? 0 : current.rightChild.sizeOfSubTree(); + } else { //go right.. + totalSmallerElements++; + current = current.rightChild; + totalSmallerElements += (current.leftChild == null) ? 0 : current.leftChild.sizeOfSubTree(); + } + } + return current; +} +{% endhighlight %} +

Here the sizeOfSubTree method just returns one plus the number of children values of the node. The totalSmallerElements variable effectively stores the index of the current node is maintained in lines as the tree is traversed.

+

Doing without recursionYou might have noticed that the code so far has been iterative rather than recursive. Generally, most operations involving trees are written using recursion, but since iterative solutions tend to be quicker, I've stuck to using iteration throughout the implementation (the only exception is with the structurallyEqualTo method which is just there for testing). For methods where you just need to traverse the tree, like the get or contains methods, turning it from a recursive method to a iterative one is just a case of adding a while loop and keeping a reference to the current Node that you're looking at. For example, you go from something like:

+{% highlight java %} +void method(){ + if(/*some condition holds for this node*/){ + return this; + } else if(/*need to traverse left*/){ + return leftChild.method(); + } else { //need to traverse right.. + return rightChild.method(); + } +} +{% endhighlight %} +

to something like:

+{% highlight java %} +void method(){ + Node currentPosition = this; + while(currentPosition != null){ + if(/*some condition holds for currentPosition*/){ + return currentPosition; + } else if(/*need to traverse left*/){ + currentPosition = leftChild; + } else { //need to traverse right.. + currentPosition = rightChild; + } +} +{% endhighlight %} +

The only difficulty comes when the method needs to go back to nodes that have previously been visited, (i.e. those that can't be written with just simple tail recursion). For instance, if you want to print all the elements in the tree in order; with recursion this is just a few lines:

+{% highlight java %} +void printAll(){ + if(leftChild != null){ + leftChild.printAll(); + } + printValues(); //prints the values at this node.. + if(rightChild != null){ + rightChild.printAll(); + } +} +{% endhighlight %} +

This could then be invoked on the root node to print the whole tree. It's really not obvious how to do this without using recursion! To overcome this, the Node class in the implementation defines a couple of handy iterative methods - the smallestNodeInSubTree, which finds the smallest node in the sub-tree rooted at the node and successor, which find the next largest node in the tree (so returns null for the node storing the largest values in the tree). They are defined like this:

+{% highlight java %} +public Node smallestNodeInSubTree(){ + Node current = this; + while(current != null){ + if(current.leftChild == null){ + break; + } else { + current = current.leftChild; + } + } + return current; +} + +public Node successor(){ + Node successor = null; + if(rightChild != null){ + successor = rightChild.smallestNodeInSubTree(); + } else if(parent != null){ + Node current = this; + while(current != null && current.isRightChildOfParent()){ + current = current.parent; + } + successor = current.parent; + } + return successor; +} +{% endhighlight %} +

With these in place, you could write an iterative version of the printAll method like this:

+{% highlight java %} +void printAll(){ + Node current = this.smallestNodeInSubTree(); + while(current != null){ + current.printValues(); //prints the values at the current node.. + current = current.successor(); + } +} +{% endhighlight %} +

Unit TestsI always find that when writing code like this, it's really easy to make a mistake, so to build up some confidence in it I wrote some junit tests for the SortedList class which are included in the download. If you find a problem with it that's not covered by them please let me know.

+

Mark Rhodes

diff --git a/_posts/2018-11-09-7-reasons-i-love-open-source-copy.md b/_posts/2018-11-09-7-reasons-i-love-open-source-copy.md new file mode 100644 index 0000000000..e8c23d855c --- /dev/null +++ b/_posts/2018-11-09-7-reasons-i-love-open-source-copy.md @@ -0,0 +1,30 @@ +--- +title: 7 Reasons I ❤️ Open Source +date: 2018-11-09 00:00:00 Z +categories: +- ceberhardt +- Open Source +author: ceberhardt +layout: default_post +summary: Here's why I spend so much of my time—including evenings and weekends—on + GitHub, as an active member of the open source community. +canonical_url: https://opensource.com/article/18/11/reasons-love-open-source +--- + +Here's why I spend so much of my time—including evenings and weekends—[on GitHub](https://github.com/ColinEberhardt/), as an active member of the open source community. + +I’ve worked on everything from solo projects to small collaborative group efforts to projects with hundreds of contributors. With each project, I’ve learned something new. + +That said, here are seven reasons why I contribute to open source: + + - **It keeps my skills fresh.** As someone in a management position at a consultancy, I sometimes feel like I am becoming more and more distant from the physical process of creating software. Working on open source projects allows me to get back to what I love best: writing code. It also allows me to experiment with new technologies, learn new techniques and languages—and keep up with the cool kids! + - **It teaches me about people.** Working on an open source project with a group of people you’ve never met teaches you a lot about how to interact with people. You quickly discover that everyone has their own pressures, their own commitments, and differing timescales. Learning how to work collaboratively with a group of strangers is a great life skill. + - **It makes me a better communicator.** Maintainers of open source projects have a limited amount of time. You quickly learn that to successfully contribute, you must be able to communicate clearly and concisely what you are changing, adding, or fixing, and most importantly, why you are doing it. + - **It makes me a better developer.** There is nothing quite like having hundreds—or thousands—of other developers depend on your code. It motivates you to pay a lot more attention to software design, testing, and documentation. + - **It makes my own creations better.** Possibly the most powerful concept behind open source is that it allows you to harness a global network of creative, intelligent, and knowledgeable individuals. I know I have my limits, and I don’t know everything, but engaging with the open source community helps me improve my creations. + - **It teaches me the value of small things.** If the documentation for a project is unclear or incomplete, I don’t hesitate to make it better. One small update or fix might save a developer only a few minutes, but multiplied across all the users, your one small change can have a significant impact. + - **It makes me better at marketing.** Ok, this is an odd one. There are so many great open source projects out there that it can feel like a struggle to get noticed. Working in open source has taught me a lot about the value of marketing your creations. This isn’t about spin or creating a flashy website. It is about clearly communicating what you have created, how it is used, and the benefits it brings. + +I could go on about how open source helps you build partnerships, connections, and friends, but you get the idea. There are a great many reasons why I thoroughly enjoy being part of the open source community. + +You might be wondering how all this applies to the IT strategy for large financial services organizations. Simple: Who wouldn’t want a team of developers who are great at communicating and working with people, have cutting-edge skills, and are able to market their creations? \ No newline at end of file diff --git a/_posts/2019-04-18-cloud-as-a-value-driver-copy.md b/_posts/2019-04-18-cloud-as-a-value-driver-copy.md new file mode 100644 index 0000000000..1b8925d54a --- /dev/null +++ b/_posts/2019-04-18-cloud-as-a-value-driver-copy.md @@ -0,0 +1,31 @@ +--- +title: 'White Paper: Thinking differently - the cloud as a value driver' +date: 2019-04-18 00:00:00 Z +categories: +- ceberhardt +- Resources +tags: +- featured +summary: The Financial Services industry is having to change and adapt in the face + of regulations, competition, changes in buying habits and client expectations. This + white paper encourages the industry to look at public cloud not as a tool for driving + down costs, but as a vehicle for technical and business agility. +author: ceberhardt +image: ceberhardt/assets/featured/cloud-value-driver.png +cta: + link: http://blog.scottlogic.com/ceberhardt/assets/white-papers/cloud-as-a-value-driver.pdf + text: Download the White Paper +layout: default_post +--- + +The Financial Services industry is having to change and adapt in the face of regulations, competition, changes in buying habits and client expectations. Technology is central to many of these changes, and in order to respond quickly it must be an enabler, not an inhibitor. + + + +One of the greatest technology enablers of the past decade is public cloud. The strategic importance of this has been widely accepted by the industry, however, the prevailing focus on the cloud as a means to reduce costs, is overlooking its greatest capability: agility! + +Public cloud platforms give an unprecedented level of technical agility. Their pay-as-you-go model makes it easy to experiment with and evaluate different technology solutions, and the high levels of automation allow rapid iteration and feedback. The cost-effective scalability of the cloud allows you to easily create systems that provision extra capacity in real-time. Furthermore the effort and cost required to make cloud solutions scalable, secure and robust is greatly reduced. + +The public cloud provides a platform for change, and a foundation for business agility. It allows you to create new services, experiment with new technology, explore SaaS offerings and provide greater user engagement with a rapid time-to-market. + +If you are interested in reading more, download the white paper: ["Thinking differently - the cloud as a value driver" - in PDF format](https://go.scottlogic.com/thinking-differently). \ No newline at end of file diff --git a/_posts/2023-02-07-state-of-open-con-copy.md b/_posts/2023-02-07-state-of-open-con-copy.md new file mode 100644 index 0000000000..96ea93336e --- /dev/null +++ b/_posts/2023-02-07-state-of-open-con-copy.md @@ -0,0 +1,24 @@ +--- +title: Could the Public Sector Solve the OSS Sustainability Challenges? +date: 2023-02-07 00:00:00 Z +categories: +- ceberhardt +- Tech +summary: The rapid rise in the consumption or usage of open source hasn’t been met + with an equal rise in contribution – to put it simply, there are far more takers + than givers, and the challenges created by this imbalance are starting to emerge. +author: ceberhardt +video_url: https://www.youtube.com/embed/aW-gVidiQsg +short-author-aside: true +image: "/uploads/Could%20PS%20solve%20the%20OSS%20Sustainability%20Challenges.png" +layout: video_post +--- + +The rapid rise in the consumption or usage of open source hasn’t been met with an equal rise in contribution – to put it simply, there are far more takers than givers, and the challenges created by this imbalance are starting to emerge. + +Most industries turn to open source for innovation and collaboration, however, the public sector instead looks for transparency and productivity. Public sector organisations have well-intentioned open source software policies, but they fail to embrace the broad potential value of open source. + +In this talk we’ll take a data-driven approach to highlight the needs of public sector organisations and explore potential opportunities. Finally, we’ll look at how this sector might be the key to solving OSS’ sustainability challenges for the long term. + +![state of opencon](/ceberhardt/assets/04-Could-the-Public-sector-solve-OSS-sustainability-challenges.png) + diff --git a/_posts/2023-04-03-beyond-the-hype-y2q-the-end-of-encryption-as-we-know-it-copy.markdown b/_posts/2023-04-03-beyond-the-hype-y2q-the-end-of-encryption-as-we-know-it-copy.markdown new file mode 100644 index 0000000000..a997f63382 --- /dev/null +++ b/_posts/2023-04-03-beyond-the-hype-y2q-the-end-of-encryption-as-we-know-it-copy.markdown @@ -0,0 +1,44 @@ +--- +title: 'Beyond the Hype: Y2Q – The end of encryption as we know it?' +date: 2023-04-03 09:00:00 Z +categories: +- Podcast +tags: +- Quantum Computing +- Y2Q +- encryption +- cryptography +- random number generation +- Security +- data security +summary: In this episode – the second of a two-parter – we talk to Denis Mandich, + CTO of Qrypt, about the growing threat that Quantum Computers will ultimately render + our current cryptographic techniques useless – an event dubbed ‘Y2Q’, in a nod to + the Y2K issue we faced over twenty years ago. +author: ceberhardt +image: "/uploads/BeyondTheHype%20-%20blue%20and%20orange%20-%20episode%2011%20-%20social.png" +--- + + + +In this episode – the second of a two-parter – Oliver Cronk and I talk to Denis Mandich, CTO of Qrypt, a company that creates quantum-secure encryption products. + +Our conversation covers the perils of bad random number generation, which undermines our security protocols, and the growing threat that Quantum Computers will ultimately render our current cryptographic techniques useless – an event dubbed ‘Y2Q’, in a nod to the Y2K issue we faced over twenty years ago. + +Missed part one? You can [listen to it here](https://blog.scottlogic.com/2023/03/13/beyond-the-hype-quantum-computing-part-one.html). + +Links from the podcast: + +* [Qrypt](https://www.qrypt.com/) – the company where Denis is CTO + +* [A 'Blockchain Bandit' Is Guessing Private Keys and Scoring Millions](https://www.wired.com/story/blockchain-bandit-ethereum-weak-private-keys/) + +* [Y2Q: quantum computing and the end of internet security](https://cosmosmagazine.com/science/y2q-quantum-computing-and-the-end-of-internet-security/) + +You can subscribe to the podcast on these platforms: + +* [Apple Podcasts](https://podcasts.apple.com/dk/podcast/beyond-the-hype/id1612265563) + +* [Google Podcasts](https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5saWJzeW4uY29tLzM5NTE1MC9yc3M?sa=X&ved=0CAMQ4aUDahcKEwjAxKuhz_v7AhUAAAAAHQAAAAAQAQ) + +* [Spotify](https://open.spotify.com/show/2BlwBJ7JoxYpxU4GBmuR4x) diff --git a/_posts/2023-09-19-metrics-collector-in-jest-copy.md b/_posts/2023-09-19-metrics-collector-in-jest-copy.md new file mode 100644 index 0000000000..bdb6ad7c43 --- /dev/null +++ b/_posts/2023-09-19-metrics-collector-in-jest-copy.md @@ -0,0 +1,222 @@ +--- +title: Optimizing Test Suite Metrics Logging in Jest Using `metricsCollector` +date: 2023-09-19 12:00:00 Z +categories: +- Testing +- Tech +tags: +- testing +- jest +summary: Discover how to streamline metrics collection in Jest test suites using a + centralized 'metricsCollector' utility, simplifying test maintenance and enhancing + data-driven testing practices. +author: gsingh +image: "/uploads/optimising%20test%20suite%20metrics.png" +--- + +When striving for robust code quality, efficient testing is non-negotiable. Logging metrics from your test suite can provide valuable insights into the performance and reliability of your codebase. In this blog post, we'll explore a resourceful method to log metrics effectively in Jest test suites using the `metricsCollector` module. This approach not only keeps your codebase clean and efficient but also allows you to seamlessly incorporate metrics recording into your testing process. + +## The Hypothesis + +Let's set the stage with a hypothetical scenario: You're developing an application that relies on an API. This API call, while essential for your application, is notorious for its carbon footprint. It returns a value containing the amount of CO2 emitted during the call. With an eco-conscious mindset, you're eager to quantify the environmental impact of your software testing. Your goal is to measure the total CO2 emissions during your test runs, not just to validate your code. + +## The Naive Approach + +Before we delve into the solution, consider the naive approach. +Here's an example of a test file (co2EmissionsNaive.test.js) using the naive approach without the metricsCollector module. This example demonstrates what the code might look like when metrics logging is managed manually inside a test suite: + +~~~javascript +//co2EmissionNaive.test.js + +const environmentallyUnfriendlyAPI = require("../test-utils/mocks/apiMock"); // This is our function to call the APIs +const co2Metrics = require("../test-utils/metrics/calculateCO2Metrics"); // This is our function which has all our calculations for the CO2 emisions. + +describe("Testing the API Calls - Naive Approach", () => { + let suiteMetrics = []; + let singleCO2Emissions = 0; + + afterAll(() => { + const { totalCO2Emissions, meanCO2Emissions } = co2Metrics(suiteMetrics); // Returns the totalCO2Emissions and meanCO2EMissions using the suiteMetrics. + console.log("Total CO2 emissions for the suite", totalCO2Emissions); + console.log("Mean CO2", meanCO2Emissions); + }); + + afterEach(() => { + const metrics = { CO2Emissions: singleCO2Emissions }; + + // Pushing the metrics that we want to record + suiteMetrics.push(metrics); + }); + + test("Test the API call with 10", async () => { + // Make the environmentally unfriendly API call + const result = await environmentallyUnfriendlyAPI(10); + + // Record the CO2 emissions metric + singleCO2Emissions = result.data.CO2Emissions; + + // Ensure that the result is as expected + expect(result.data.output).toBe(true); + }); + + test("Test the API call with 15", async () => { + const result = await environmentallyUnfriendlyAPI(15); + singleCO2Emissions = result.data.CO2Emissions; + expect(result.data.output).toBe(true); + }); +}); +~~~ + +When the test is run, it produces the below result + +![Mean and total CO2 Emissions are logged in the console]({{site.github.url}}/gsingh/assets/./naiveResult.PNG "Mean and total CO2 Emissions are logged in the console") + +If we have multiple test suites where we are using this environmentallyUnfriendlyAPI calls and want to log their CO2 Emission data, then you could copy-paste metric recording and logging code into each test file. This approach clutters your test files, making them harder to read and maintain. It's prone to inconsistencies, and calculating suite-level or overall metrics becomes a complex, error-prone task. Let's be honest; this approach is neither clean nor efficient. + +## The Metrics Collector Solution + +The solution lies in the metricsCollector module. This custom module streamlines metrics collection and management within your test suites, eliminating the need for repetitive code. Here's how it works: + +~~~javascript +// metricsCollector.js + +const metricsCollector = () => { + let metrics = {}; // store a single metric + let suiteMetrics = []; // Store suite-level metrics + + // This function is used to record the metric + const recordMetric = (key, value) => { + metrics[key] = value; + }; + + const clearMetrics = () => { + metrics = {}; + }; + + // This function is used to return the suite Metrics + const getSuiteMetrics = () => { + return suiteMetrics; + }; + + // This function is used to add a single test's metrics to the suite metrics + const addToAllMetrics = () => { + suiteMetrics.push(metrics); + }; + + // This function is used to console log all the suite metrics + const logMetrics = () => { + suiteMetrics.forEach((m) => { + for (const key in m) { + console.log(`Logging metrics -- ${key}: ${m[key]}`); + } + }); + }; + + // beforeEach jest hook, here we are clearing the test level metrics before running the next test + beforeEach(async () => { + clearMetrics(); + }); + + // afterEach jest hook, here we are adding a single test's metrics to the suite level before running the next test. + afterEach(async () => { + addToAllMetrics(); + }); + + // Here we are exposing all the functions that we think can be used in the test suites to use the suite metrics. + return { recordMetric, logMetrics, getSuiteMetrics }; +}; + +module.exports = metricsCollector; +~~~ + +In this solution: + +- metricsCollector initializes metric storage. +- Metrics are recorded at both the test case and suite levels. +- It simplifies logging and provides flexibility in calculating suite-level metrics. +- If we want to include more functions in our metricsCollector module around suiteMetrics, we can have those and then can use those functions in our test suites. + +## Integration into Test Suites + +Now, let's see how you use it in your sample test suite, co2EmissionModule.test.js: + +~~~javascript +// co2EmissionModule.test.js + +const environmentallyUnfriendlyAPI = require("../test-utils/mocks/apiMock"); +const co2Metrics = require("../test-utils/metrics/calculateCO2Metrics"); +const metricsCollectorModule = require("../test-utils/metricsCollector"); + +const { recordMetric, getSuiteMetrics, logMetrics } = metricsCollectorModule(); // This will return the functions e.g. recordMetric, getSuiteMetrics + +describe("Testing the API Calls - Naive Approach", () => { + afterAll(async () => { + const suiteMetrics = getSuiteMetrics(); // Returns all the metrics collected for this test suite. + const { totalCO2Emissions, meanCO2Emissions } = co2Metrics(suiteMetrics); // Returns the totalCO2Emissions and meanCO2EMissions using the suiteMetrics. + console.log("Total CO2 emissions for the suite", totalCO2Emissions); + console.log("Mean CO2", meanCO2Emissions); + }); + + test("Test the API call with 10", async () => { + // Make the environmentally unfriendly API call + const result = await environmentallyUnfriendlyAPI(10); + + // Record the CO2 emissions metric + recordMetric("CO2Emissions", result.data.CO2Emissions); + + // Ensure that the result is as expected + expect(result.data.output).toBe(true); + }); + + // ... (similar tests follow) +}); +~~~ + +#### _Test results_ + +When the test is run, it produces the below result + +![Mean and total CO2 Emissions are logged in the console]({{site.github.url}}/gsingh/assets/moduleResult.PNG "Mean and total CO2 Emissions are logged in the console") + +By using this modularised approach, if we want to use 'logMetrics' function in another test suite, we can just plug it in our afterAll hook and it will work as the following. + +~~~javascript +//co2EmissionModule.test.js + +// previous import statements + +const { logMetrics } = metricsCollectorModule(); // This will return the functions e.g. recordMetric, getSuiteMetrics, logMetrics + +describe("Testing the API Calls - Naive Approach", () => { + afterAll(async () => { + logMetrics(); // Plugging logMetrics + }); + + test("Test the API call with 10", async () => { + // Make the environmentally unfriendly API call + const result = await environmentallyUnfriendlyAPI(20); + + // Record the CO2 emissions metric + recordMetric("CO2Emissions", result.data.CO2Emissions); + + // Ensure that the result is as expected + expect(result.data.output).toBe(true); + }); + + // ... (similar tests follow) +}); +~~~ + +When the test is run, it produces the below result + +![Metrics are logged]({{site.github.url}}/gsingh/assets/moduleLogMetrics.PNG "Metrics are logged") + +## The Results and Conclusion + +In this blog post, we've tackled the challenge of tracking environmental impact in your Jest test suites. We started with a scenario where an environmentally unfriendly API call produces CO2 emissions. We contrasted a naive approach, which involves repetitive metric tracking in each test file, with a more streamlined approach using the metricsCollector. + +By centralizing metrics tracking, you can keep your test files clean and maintainable, while also gaining the flexibility to log metrics at different levels. With our metricsCollector module seamlessly integrated, running our test suite yields insightful metrics logging without cluttering the test code itself. The common module approach centralizes metrics management, promoting clean and focused tests. + +In conclusion, our hypothesis was successfully tested and validated. By leveraging the metricsCollector module, we achieved a streamlined and organised way to log metrics during Jest test executions. This method enhances the maintainability and readability of our test suite, enabling us to focus on what matters most: writing high-quality, well-tested code. + +_Note: This blog post provides a high-level overview of logging metrics in Jest test suites. For more advanced use cases and in-depth analysis, you can extend the metrics collector and data processing logic to suit your specific needs_. diff --git a/_posts/2023-10-18-the-state-of-webassembly-2023-copy.markdown b/_posts/2023-10-18-the-state-of-webassembly-2023-copy.markdown new file mode 100644 index 0000000000..f53fabdb58 --- /dev/null +++ b/_posts/2023-10-18-the-state-of-webassembly-2023-copy.markdown @@ -0,0 +1,166 @@ +--- +title: The State of WebAssembly 2023 +date: 2023-10-18 16:01:00 Z +categories: +- Tech +summary: This blog posts shares the results of the third annual State of WebAssembly + survey, where we found that Rust and JavaScript usage continues to increase, but + there is a growing desire for Zig and Kotlin. The use of wasm as a plugin environment + continues to climb, with developers hoping it will deliver of the “write once and + run anywhere” promise. +author: ceberhardt +image: "/uploads/state%20of%20web%20assembly%202023.png" +--- + +The State of WebAssembly 2023 survey has closed, the results are in … and they are fascinating! + +If you want the TL;DR; here are the highlights: + +* Rust and JavaScript usage is continuing to increase, but some more notable changes are happening a little further down - with both Swift and Zig seeing a significant increase in adoption. +* When it comes to which languages developers ‘desire’, with Zig, Kotlin and C# we see that desirability exceeds current usage +* WebAssembly is still most often used for web application development, but serverless is continuing to rise, as is the use of WebAssembly as a plugin environment. +* Threads, garbage collection and the relatively new component model proposal, are the WebAssembly developments that people are most interested in. +* Whereas with WASI, it is the I/O proposals (e.g. HTTP, filesystem) that garner the most attention. +* We are potentially seeing some impatience in the community, with the satisfaction in the evolution of WASI being notably less than the satisfaction people express in the evolution of WebAssembly. +* Many respondents shared that they expect WebAssembly to deliver on the “write once and run anywhere” promise that was originally made by Java. + +(If you want to look back, here are the [2021](https://blog.scottlogic.com/2021/06/21/state-of-wasm.html) and [2022](https://blog.scottlogic.com/2022/06/20/state-of-wasm-2022.html) results) + +Interested to learn more? Then read on … + +## Language + +The first question explored which languages people are using by asking the question _which languages do you use, or have you tried using, when developing applications that utilise WebAssembly?_ + +![wasm-language-usage.png](/uploads/wasm-language-usage.png) + +For the third year running, Rust is the most frequently used language for WebAssembly. Rust has always been a good fit for WebAssembly; it is a modern system-level language that has broad popularity (the Stack Overflow revealed it is the most desired language seven years in a row), it also happens to be a popular language for authoring WebAssembly runtimes and platforms. + +JavaScript is the second most widely used language, which is quite notable considering that you cannot compile JavaScript to WebAssembly. To run JavaScript code, the runtime is compiled to WebAssembly, with your code running within the WebAssembly-hosted interpreter. This approach, which might sound inefficient, is surprisingly practical and increasingly popular. You may not get a speed advantage, but do benefit from the security and isolation benefits of WebAssembly. For further details, I’d recommend this [in-depth article from the Shopify team](https://shopify.engineering/javascript-in-webassembly-for-shopify-functions) which describes how they support ‘Shopify functions’ written in JavaScript, which run on a WebAssembly platform. + +The following chart shows the long-term trends, comparing the results from the last three surveys, with the percentage of people using each language (frequently or sometimes) - excluding those with <10% usage. + +![wasm-language-usage-trends.png](/uploads/wasm-language-usage-trends.png) + +Usage of Rust and JavaScript is increasing, but some more notable changes are happening a little further down. Both Swift and Zig have seen a significant increase in adoption. + +Swift is a relatively recent addition to the WebAssembly ecosystem, starting a few years ago with a [pull request on Apple’s Swift repo](https://github.com/apple/swift/pull/24684) to add a wasm target. However, despite having numerous commits over many years, this PR hasn’t been merged. It looks like the community is undeterred and are [maintaining their own fork](https://swiftwasm.org/). + +While Swift and Rust are both quite new languages (2014 and 2015 respectively), Zig is even younger, having emerged in 2016, which makes it one year older than WebAssembly (which had its first MVP release in 2017). + +This year I added a new question to the survey which asked _what is your professional relationship with WebAssembly?_ With a goal of separating responses from people who are actively developing WebAssembly tools, or platforms and those who are simply end users. Separating these two groups, we see the following language preferences: + +![wasm-language-use-end-user.png](/uploads/wasm-language-use-end-user.png) + +As expected, tool developers have a strong preference for Rust, and also enjoy programming WebAssembly directly using WAT (WebAssembly Text Format). There is also a strong preference for Go and Python - which is something I wasn’t expecting. + +The next question in the survey explored how desirable each language is by asking the question _which languages do you want to use in the future to develop applications that utilise WebAssembly?_ + +![wasm-language-desire.png](/uploads/wasm-language-desire.png) + +Once again, Rust comes out top, reflecting the findings of the annual Stack Overflow survey, with JavasScript in second. However, Zig, which is somewhat infrequently used, is the third most desired language. + +Plotting the delta for each language, between the number of “frequently used” responses and “want to use a lot”, for desirability, we can see which languages have the biggest difference in desirability vs. usage: + +![wasm-desire-vs-use.png](/uploads/wasm-desire-vs-use.png) + +At one end of the spectrum, Zig, Kotlin and C# we see that desirability exceeds current usage, whereas at the other end, people would prefer to use less C++, JavaScript and WAT. + +## Runtime + +Considering that non-browser based usage of WebAssembly on the climb, it’s interesting to explore which runtimes people are using, or are simply aware of, the survey simply asked _which have you heard about or used?_ + +![wasm-runtime-usage.png](/uploads/wasm-runtime-usage.png) + +[wasmtime](https://github.com/bytecodealliance/wasmtime), from Bytecode Alliance is the most widely used, with [wasmer](https://wasmer.io/), which is developed by a start-up, coming in second. [Wazero](https://wazero.io/) is a new addition to the list, a recently released runtime built in Go. + +## Practical applications of WebAssembly + +The survey asked _what are you using WebAssembly for at the moment?_, allowing people to select multiple options and add their own suggestions. Here are all of the responses, with ‘Other’ including everything that only has a single response: + +![wasm-usage-update.png](/uploads/wasm-usage-update.png) + +Web application development is still at the top, but the gap has closed a little. The following chart reveals the year-on-year trends: + +![wasm-usage-trend-update.png](/uploads/wasm-usage-trend-update.png) + +NOTE: In the 2021 / 2022 surveys, 'Serverless' was the only option for back-end usage of wasm. In 2023 this has been split into two distinct categories, hence the dotted line for Serverless in the above chart. Combining the two options from 2023 would show a minor increase in back-end usage. + +The most notable shift is the use of WebAssembly as a plugin environment. Here are some real-world examples: + +* Zellij is a developer-focussed terminal workspace that has a [WebAssembly plugin model](https://zellij.dev/news/new-plugin-system/) +* Microsoft Flight Simulator allows you to [write add-ons as wasm modules](https://docs.flightsimulator.com/html/Programming_Tools/WASM/WebAssembly.htm) +* Envoy and Istio have a [Wasm Plugin API](https://istio.io/latest/blog/2021/wasm-api-alpha/) +* Lapce, a new IDE written in Rust, has a [WASI-based plugin system](https://lapce.dev/) + +In each case, the platform (terminal, editor, flight simulator, proxy) benefits from allowing end-users to extend the functionality, using a wide range of programming languages, in an environment that is safe and isolated. In other words, if someone writes a plugin that misbehaves, or simply has poor performance, the impact on the platform itself is minimised. + +We also asked respondents - _what’s the status of your organisation’s WebAssembly adoption?_ + +![wasm-org-usage.png](/uploads/wasm-org-usage.png) + +From the above chart we can see that 41% of respondents are using WebAssembly in production, with a further 28% piloting or planning to use it in the next year. + +The survey also explored what WebAssembly needs to help drive further adoption: + +![wasm-needs.png](/uploads/wasm-needs.png) + +The most frequently cited ‘need’ was better non-browser integration, through WASI (WebAssembly System Interface). The WebAssembly specification doesn’t define any host integration points, whether this is how you access the DOM, or exchange data with the host runtime (e.g. pass values to JavaScript within the browser). WASI is plugging this gap, but doesn’t have a complete answer just yet. + +Better debugging support is a very close second, which will become more important as people develop more complex solutions with WebAssembly. For a good overview options, check out [this blog post from the Shopify team](https://shopify.engineering/debugging-server-side-webassembly). + +## Features, features, features + +Both WebAssembly (which is managed by W3C) and WASI (managed by a sub-organization of the WebAssembly Community Group of the W3C) are constantly evolving, with a backlog of new features that follow the standard 5-phase proposal process. + +Regarding WebAssembly proposals, the following shows which are the most desired: + +![wasm-feature-desire.png](/uploads/wasm-feature-desire.png) + +Threads, garbage collection and exception handling were all at the top in last year's results, and all three are at implementation (phase 3) or standardisation (phase 4) in the proposal lifecycle. This means they are ready to use, and close to finalisation. + +Component model is a much more early-stage proposal (phase 1), with a broad ambition to make it much easier to compose wasm modules, written in any language, at runtime. If you’re interested in the details, I’d recommend this [video from Luke Wagner](https://www.youtube.com/watch?v=tAACYA1Mwv4), who is leading on the proposal. + +Regarding WASI proposals, the following shows which are the most desired: + +![wasi-feature-desire.png](/uploads/wasi-feature-desire.png) + +The four top proposals are all I/O related, quite simply, creating a standard way for WebAssembly modules to communicate with the outside world is a priority. + +Finally, we asked how satisfied people are with the evolution of WebAssembly and WASI: + +![wasm-wasi-satisfaction.png](/uploads/wasm-wasi-satisfaction.png) + +There are a significant number of people who are not satisfied! This isn’t at all surprising, evolving specifications, that have so many stakeholders, in an open and transparent fashion is not easy and takes time. What is probably more notable is that generally speaking, people are less satisfied with the evolution of WASI. + +I do want to make an important point here; this result should not be used as a direct criticism of the fantastic efforts the WASI and WebAssembly groups are making. The lack of satisfaction in the evolution of WASI could simply be a reflection of the eagerness people have for the technology, which is not a bad thing. + +Earlier this year Wasmer announced [WASIX](https://wasmer.io/posts/announcing-wasix), which is their attempt to accelerate WASI (or the concepts it represents), to a mixed response. + +## And finally + +I asked people _what is the thing that excites you most about WebAssembly?_ And almost half the respondents shared their thoughts, far more than I can realistically reproduce here. So, I did the most sensible thing, I asked ChatGPT to summarise the key themes: + + + * Portability and the ability to run code on different platforms + * Interoperability between different languages and the web + * Native performance and efficiency + * Access to existing code and libraries + * The potential for new languages and tools + * Security and sandboxing capabilities + * The ability to replace containers and run complex stacks in the browser + * The potential for a universal binary format + * The opportunity to write once and run anywhere + * Improved performance and speed + * The component model and the ability to reuse code + * The reduction or elimination of JavaScript dependence + * More flexibility and choice in language selection + * The potential for a plugin system + * The potential for running complex applications in the browser + +Thank you to everyone who shared their thoughts, much appreciated. + +If you want to explore the data, feel free to [download the dataset](https://wasmweekly.news/assets/state-of-webassembly-2023.csv), please do attribute if you reproduce or use this data. You can also [discuss this post over on Reddit](https://www.reddit.com/r/programming/comments/17ax4ek/the_state_of_webassembly_2023/). + +Finally, I want to thank [Lawrence Hecht](https://www.linkedin.com/in/lawrence-hecht/), who I've worked with on a few survey / research projects previously, for his feedback on the 2023 survey. Very much appreciated! +