Map

This is an automatically generated graph containing all pages on my websites, along with the connections calculated using sentence embeddings. If you're interested, you can read the source code.

How is this thing generated?

1. All of the posts are fed through an embeddings generator, I'm using the Sentence Transformers Python library.
2. The embeddings are passed to UMAP, a dimensionality reduction algorithm, which takes in multi-dimensional embeddings and projects it down to a 2D representation, which can be drawn as a graph. The projection is done so that the high-level "structure" of the data is preserved.
3. I'm connecting each post with its top 2 nearest posts (using more clutters up the map).
4. graphviz renders the graph and outputs it as an SVG file.

A much better description on Simon Willison's blog.

Future plans

1. Make this thing look more "map-alike", whatever that might mean.
2. Experiment with text clusterisation & dimensionality reduction algorithms, such as:
   • tSNE
   • K-means clustering
   • UMAP
   • Latent Dirichlet allocation
   • DBSCAN
3. Add #tags.
4. Introduce color-coding and other visual markers, allowing viewers to make sense of the data based on different metrics:
   • Post tags
   • Links to/from other posts
   • Links outside (to the netsphere)
   • Other connections generated by NLP
5. Check out KagiSearch/vectordb