Overview
SkeleTrace is an engine for open-source investigation. It keeps entities, the links between them and measurements over time, and builds views of them when asked. On top of that model sit four tools:
- weak points: the few nodes and links a group depends on;
- the strongest chain: the most believable route between two nodes;
- geofences: which nodes fall inside an area, and how much ground a set of nodes covers;
- text sweeps: every watch-list term in any number of files, and words a typo or two away.
Everything runs from one command-line program, skeletrace, on Windows, macOS and Linux. Every command answers in JSON, and the viewer draws much of it. Each file is described field by field on the data formats page.
SkeleTrace Community is free software, under the GNU General Public License, version 3. It can be downloaded ready to run, or built from its source.
The model
The engine keeps the smallest true picture it can, and builds everything else from it when asked.
- Entities
- Nodes and the edges, or links, between them, each with a stable identifier, a kind, a status and a priority. Each also has tags, and the times it was first and last seen. Node kinds include endpoints, relays, exchanges, observers, identities, accounts and topics. Edge kinds include routes, links, inferences, associations, memberships, references and boundary crossings. A node may carry a position, and every link carries a confidence from 0 to 1.
- Boundaries
- Named areas, such as a jurisdiction, a network zone or a platform, with a rectangular extent where they have a place on the map.
- Measurements
- Metric definitions with units, and samples recorded against them as an append-only series. Each metric has a retention policy, so history is kept for as long as it is useful and no longer.
- Working set
- A small cache of what is active, selected or on screen, held within a fixed budget. Everything else stays in storage until it is needed.
- Views
- Topology views, sparse geographic views, data cards and snapshots, produced on request. A view is a projection of the stored record at a moment, not a second copy of it.
The engine does not try to hold a model of the whole world in memory. It holds what has been observed, and answers questions about it.
A profile, a JSON file, holds the engine’s settings and its metrics, nodes, edges, boundaries and sources. Most commands take one as their first argument.
Sources and provenance
Data reaches the engine through adapters: NDJSON sample files, HTTP JSON endpoints, polled feeds, HTTP over Tor and manual pushes. Two more read Lookyloo, the web capture tool. One takes its capture summaries as measurements; the other builds a graph of captures, domains and redirects.
Every sample keeps its provenance: the source, the adapter that fetched it, the address it came from and when it was retrieved. A governance layer records each source’s policy and capabilities, and sorts failures into kinds. An audit trail lists the samples taken and the snapshots written.
Sources are polled on a schedule, and a source that fails waits longer before each new try.
Weak points
A chokepoint is the one bridge into a valley town: close it, and part of the map is cut off. analyze-topology looks for these in any topology view.
- Group
- A set of nodes joined to each other by some path. Groups are listed largest first.
- Chokepoint
- A node whose removal splits its group, reported with the number of pieces it leaves behind.
- Bridge
- A single link whose loss splits its group. Two parallel links between the same nodes are never bridges.
The analysis is a depth-first search after Tarjan. It runs in time proportional to the number of nodes plus links, without recursion, so very large views are handled without difficulty. In the viewer, chokepoints are ringed and bridges are drawn heavier, in every view.
In an investigation, a chokepoint can be the one source or inference on which a whole conclusion rests. That makes it the first thing to check again. Watch weak points found step by step in the lab.
The strongest chain
Every link carries a confidence from 0 to 1: how far the evidence for it can be trusted. A chain of links is only as believable as all of them together, so its confidence is their confidences multiplied.
strongest-chain finds the chain between two nodes with the highest product. Two solid links at 0.9 each, 0.81 together, beat one shaky link at 0.5.
The search is Dijkstra’s shortest-route method. Taking the negative logarithm of each confidence turns multiplying into adding, so the cheapest route is the strongest chain.
- Name each node by its identifier or its exact label. If several nodes share a label, the command lists their identifiers to choose from.
- Links count in both directions. A link at confidence 0 is never used, and the report lists it.
- When nothing joins the two nodes, the report says so. That is an answer, not an error.
- Equally strong chains are settled by link identifier, so the same view always gives the same answer.
The product is the chance that the whole chain holds, if each link holds or fails on its own. Links that rest on the same source are not independent, so read the figure as a ranking of what to check first.
Watch a strongest chain found step by step in the lab.
Geofences and covered area
fence-check reads fences saved as GeoJSON polygons, as geojson.io, QGIS and most mapping tools save them. For each fence it lists the view’s nodes inside and exactly on the edge, and gives the fence’s area in square kilometres. Holes are respected: a node in a hole is outside. Nodes in no fence, and nodes with no position, are listed separately.
Each node is placed with a crossing test. Draw a line from the node in one direction and count the fence edges it crosses: an odd number means inside.
cover-area draws the tightest outline round every node that has a position, known as the convex hull. It gives the outline’s area, and returns the outline as GeoJSON, ready for any map. Nodes all in one line enclose no area. The outline comes from Andrew’s monotone chain, which sorts the points, then sweeps along the bottom and back along the top.
Areas are worked out on an equal-area map of a sphere the size of the Earth. Edges count as straight lines on a latitude and longitude map, which suits areas up to a few hundred kilometres across. A fence that spans more than 180° of longitude is refused: split it in two.
Watch fences and outlines drawn step by step in the lab.
Text sweeps
sweep-text searches files for every term on a watch list: names, handles, email addresses or phone numbers. It needs no profile. Give it the watch list and any number of files, such as dumps, pastes, saved pages or logs.
The search is the Aho–Corasick method. It builds one machine from all the terms, then feeds it each byte of a file once. Files are read a megabyte at a time, so their size does not matter, and the number of terms barely does.
- Capital letters A to Z are ignored, unless you add
--exact. - Each hit gives the file, the byte offset, the line, the column and the text as written. Columns count characters, as text editors do.
- The report counts each term’s hits across every file, and lists the terms found nowhere. The first 10,000 hits are listed, and
--max-hitschanges that. The counts are always complete. - The watch list holds one term per line, in UTF-8 or in the UTF-16 that Windows PowerShell writes. Blank lines are skipped, and so are notes: lines starting with # and a space.
Files are searched as bytes, so UTF-8 and plain ASCII text work as they are. A file that starts with a UTF-16 byte-order mark gets a warning, because UTF-8 terms cannot match it.
Watch a sweep run step by step in the lab.
Near-misses
With --fuzzy 1 or --fuzzy 2, the same pass also lists words a typo or two away from a term: jdeo88 for jdoe88, or lantern@exmaple.org for lantern@example.org. A typo is a character added, dropped or changed, or two neighbouring characters swapped.
Short terms sit one typo away from too many ordinary words, so the allowance grows with length.
| Term length | Typos allowed |
|---|---|
| Under 5 characters | None: exact matches only |
| 5 to 7 characters | 1 |
| 8 characters or more | 2, or 1 with --fuzzy 1 |
For this, the text is split into words: runs of letters and digits in any alphabet, plus @ . _ - + #. A word with an @ in it is also tried piece by piece, so a handle inside an email address is found. Terms with spaces, such as a spaced phone number, are only matched exactly.
Two strings a typo or two apart can always be cut down to the same string by deleting a character or two from each. So each term is filed under the shorter strings its deletions leave, and each word looks up its own. Sharing a slot only makes a candidate: counting the typos directly settles it.
Near-misses come back in their own list, each with its count of typos, so the exact results read as they always did. Speeds were measured on a 300 MB made-up dump, with a 1,000-term watch list, on one core of a two-core cloud machine. Exact sweeps ran at 58 MB a second, --fuzzy 1 at 24 and --fuzzy 2 at 18. Memory stayed under 15 MB.
Watch near-misses found step by step in the lab.
The algorithms
The strongest chain, geofences, text sweeps and near-misses are built on skeletrace-algo, SkeleTrace’s own algorithm library. It is written in Rust and uses only the standard library. Weak-point analysis is part of the engine itself, and the library has its own version for other programs.
| Tool | Method | Command |
|---|---|---|
| Weak points | Tarjan’s depth-first search for cut points and bridges | analyze-topology |
| Strongest chain | Dijkstra’s shortest route, on the negative logarithm of each confidence | strongest-chain |
| Geofences | The crossing test, and an equal-area map for areas | fence-check |
| Covered area | Andrew’s monotone chain for the convex hull | cover-area |
| Text sweeps | Aho–Corasick: every term in one pass | sweep-text |
| Near-misses | Filing terms under their deletions, then a typo count that allows swapped neighbours | sweep-text --fuzzy |
The library holds more than the engine uses today: sorting and searching, heaps, range sums, union–find, string matching, edit distances and plane geometry. Many of its tests compare a function with a slow, simple method that is easy to trust.
The algorithm lab animates these functions one step at a time, on a made-up case. Each demo is a JavaScript copy of a Rust function, checked against the library’s own answers before it is published. The papers behind each method are listed under methods on the sources page.
Place and geometry
Positions are WGS 84 latitude and longitude in degrees, with an optional altitude. Nodes without a position are kept and listed, and the map views show them in a separate tray, so nothing drops out of sight. Boundaries, such as a jurisdiction or a watch zone, carry a rectangular extent given by its minimum and maximum latitude and longitude.
Sparse geographic views are produced for a requested viewport and delivered as standard GeoJSON. Every link carries a geometry mode, which decides how it is drawn:
| Mode | Drawn as |
|---|---|
Straight | A straight line on the flat map. On the globe it follows the surface. |
Geodesic | The shortest path over the Earth’s surface: a great circle. |
RaisedArc | An arc lifted above the surface, higher for longer links. |
ThroughGlobe | A dashed chord through the Earth, for links that are not routes over the surface. |
Abstract | A dotted line, for relations with no physical path. |
The viewer
The viewer draws SkeleTrace output four ways: as a 2D graph laid out by simulated forces, a 3D graph, a flat map or a globe. It opens on the map when most nodes have a position, and on the graph otherwise.
It opens topology views, sparse-geo maps and bottleneck reports. A strongest chain opens as a small graph, and a covered area as an outline. Fence checks and sweeps are lists rather than pictures, so they are read as JSON. The viewer also opens GeoJSON, GraphML, general graph JSON, CSV, KML, KMZ and GPX.
Selecting a node, link, area or track shows it in full. That means its kind, identifier and position, its metrics with units and times, every connection, and the raw data from the file. Weak points are marked in every view.
The viewer is a single page. Files opened in it are read on your computer and go nowhere else. Its content policy allows one kind of request: downloading the public map layers published here. Several files, and the built-in layers, can be shown together. Each becomes a layer with its own colour, which can be switched off or picked alone from the source menu. Coastlines are built in, from Natural Earth’s public-domain data. Files saved by Windows PowerShell redirects, which are written as UTF-16, open as they are.
A typical piece of work
- Describe. Write a profile: the nodes, edges and boundaries you know of, the metrics you track and the sources that feed them. The offline demonstration in the source code writes a complete one to copy.
- See the shape. Run
materialize-topologyormaterialize-sparse-geo, and open the output in the viewer. - Find the weak points.
analyze-topologynames the chokepoints and bridges. Check each one again, because a conclusion may rest on it. - Follow the evidence.
strongest-chaingives the most believable route between two nodes, with the confidence of every link on it. Start with its weakest link. - Place it.
fence-checksays which nodes fall inside your areas of interest, andcover-areaoutlines the ground they cover. - Sweep the text.
sweep-text --fuzzy 2finds the watch list, and its near-misses, in dumps and logs. - Keep a record.
export-snapshotwrites a view to disk as JSON, CSV or GeoJSON, with an optional catalogue.
Each command starts a fresh engine and exits, so measurements polled by one command are not seen by the next. The structure tools read the profile directly, so they work straight from the command line. To collect and query measurements over a session, keep one engine running through the Rust library.
What SkeleTrace does not do
- It does not decide who is who. It shows what is linked and how strongly. Joining two identities is a judgement for a person.
- It does not score truth. A chain’s confidence is only as good as the confidences given to its links. A near-miss is a lead to read, not a match.
- It does not run as a service. Each command does one job and exits. Collection over time belongs in your own program, through the library.
- It does not search the web for you. It reads the sources you configure and the files you give it. A case notebook, traffic analysis and movement data are separate tools: see works with.
- It does not hide failure. When a source fails, the failure is recorded with its kind, its time and the details, and the health report shows it.
Command reference
Most commands take a profile as their first argument. Every command prints one JSON document, and messages about problems go to the error stream.
| Command | What it does |
|---|---|
profile-validate PROFILE | Checks a profile and lists its nodes. |
tick PROFILE | Polls the sources that are due. |
poll-source PROFILE SOURCE | Polls one source now. |
materialize-topology PROFILE VIEW | Produces a topology view. |
materialize-sparse-geo PROFILE VIEW | Produces a geographic view as GeoJSON. |
analyze-topology PROFILE VIEW | Finds groups, chokepoints and bridges in a view. |
strongest-chain PROFILE VIEW FROM TO | Finds the most believable chain of links between two nodes. |
fence-check PROFILE VIEW FENCES | Says which nodes are inside, on the edge of, or outside each GeoJSON polygon. |
cover-area PROFILE VIEW | Outlines every node with a position, and gives the area in square kilometres and the outline as GeoJSON. |
export-snapshot PROFILE JOB DIR [CATALOGUE] | Writes a snapshot as JSON, CSV or GeoJSON, with an optional SQLite catalogue. |
query-latest PROFILE FILTER | Returns the latest values that match a filter. |
query-advanced PROFILE QUERY | Runs a saved query with grouping, sorting and comparison. |
evaluate-watchlist PROFILE WATCHLIST | Checks a watchlist’s rules against current values. |
health PROFILE | Reports on the engine, its sources, its cache and its events. |
sweep-text [--exact] [--fuzzy 1|2] [--max-hits N] TERMS FILE… | Finds every watch-list term in the files, and near-misses with --fuzzy. Needs no profile. |
Exit codes: 0 for success, 1 for an error, and 2 for a usage mistake, such as a missing argument. skeletrace --help lists every command, and skeletrace --version gives the version.