Designing Data Structures for an Audit Dashboard

This is a chapter from a running series: whatever I build for this site, I write up here as theory and as practice. This one comes from the audits tab of my private dashboard, where a report of a few thousand rows has to become something a person can read in ten seconds.

The problem

A link checker on a visa guide produces one row for every link it tried. That is thousands of rows, and most are fine. A dashboard that shows all of them answers no question. A dashboard that shows only "84 broken" answers the wrong one, because most of those 84 were not broken at all. The sites blocked an automatic check.

So the job is not "display the data". The job is to decide the shape of the data that goes to the screen. That is a data structure design problem, and the choices are the same ones you meet in any data structures course.

Theory

1. Start from the question, not the table. Write the three questions the reader will ask. For the link check they were: how many are broken, which ones do I fix first, and which ones can I ignore? Each question becomes a field in the shape. Anything that answers none of them is left out.

2. Two layers: the raw report and the view. The raw report is for machines and keeps everything. The view is a small object built from it, for people. You can always rebuild the view from the report, so the view can be thrown away and changed freely. This is the same idea as a database table and an index, or a source file and its summary.

3. Aggregate with a hash map. To group many rows by a key (the website a link points to), one pass with a Map costs O(n). Sorting the groups afterwards costs O(g log g) for g groups, and g is small. Doing it the other way, scanning all rows once for each site, costs O(n x g) and gets slow as the report grows.

4. A list that is cut short must say so. If the screen shows the top 20, send the total as well. Otherwise "20 problems" looks like the whole truth. The pair { items, total } is cheap and prevents a quiet lie.

5. Turn free text into a small set of kinds. An error arrives as "ENOTFOUND", "CERT_HAS_EXPIRED", "403", "timeout after 15000ms". A reader does not want those strings. A function that maps each to one of a few kinds (gone, server, blocked, dns, cert, slow) lets the screen sort, count and colour them, and makes the rule testable in one place.

6. Bound everything. A cap on list lengths and a cap on string lengths mean the response can never grow without limit, whatever the report contains. Bounds are part of the design, not an afterthought.

7. Contain failure. If three audits load together, one throwing must not blank the other two. Each audit returns either its data or its own error, and the screen draws whichever it got.

8. Respect the store. The database has rules too. Firestore, which I use, refuses an array placed directly inside another array. A list of rows like [['/home', 200], ['/visa', 200]] cannot be stored as is. I learned this from a failure on my own health tab. The fix is to wrap each inner array in an object on the way in and unwrap it on the way out, in one small module, so the rest of the code never knows.

Practice

Group links by site, in one pass:

function bySite(rows) {
  const groups = new Map();
  for (const r of rows) {
    const host = new URL(r.url).hostname;
    const g = groups.get(host) || { host, count: 0, countries: new Set() };
    g.count++;
    (r.countries || []).forEach((c) => g.countries.add(c));
    groups.set(host, g);
  }
  return [...groups.values()]
    .sort((a, b) => b.count - a.count)
    .map((g) => ({ ...g, countries: [...g.countries] }));
}

Turn messy errors into kinds:

function kindOf({ status, error }) {
  if (status === 404 || status === 410) return 'gone';
  if (status >= 500) return 'server';
  if (status === 403 || status === 429) return 'blocked';
  if (/ENOTFOUND/.test(error || '')) return 'dns';
  if (/CERT_HAS_EXPIRED/.test(error || '')) return 'cert';
  if (/timeout/i.test(error || '')) return 'slow';
  return 'other';
}

Build the view, with totals beside the capped lists:

function linkView(rows) {
  const bad = rows.filter((r) => !r.ok).map((r) => ({ ...r, kind: kindOf(r) }));
  const fix = bad.filter((r) => r.kind !== 'blocked');
  return {
    total: rows.length,
    broken: bad.length,
    toFix: fix.slice(0, 20),
    toFixTotal: fix.length,
    blocked: bySite(bad.filter((r) => r.kind === 'blocked')).slice(0, 10),
  };
}

Keep Firestore happy:

const WRAP = '_a';
function enc(v) {
  if (Array.isArray(v)) return v.map((x) => (Array.isArray(x) ? { [WRAP]: enc(x) } : enc(x)));
  if (v && typeof v === 'object') {
    const o = {};
    for (const k of Object.keys(v)) if (v[k] !== undefined) o[k] = enc(v[k]);
    return o;
  }
  return v;
}

Then test the shape, not the screen. Feed linkView a small made-up report and check the counts, the order and the totals. A test that pretends the database accepts anything is the one that lets a storage rule slip through, which is exactly what happened to me. Make the fake database refuse what the real one refuses.

Try it yourself

1. Take any CSV of events (orders, logins, page views). Write down three questions a reader would ask of it.
2. Design the smallest object that answers them. Add totals next to any list you cut short.
3. Write the one-pass Map grouping, then the kind function for any free-text column.
4. Write a test that checks the counts on a ten-row input.

What to take from this

The data structure is the design. Decide the questions, keep a raw layer and a view layer, aggregate in one pass, say when a list is cut short, turn text into kinds, put bounds on everything, and test against a store that is as strict as the real one.

← Back to Guidance