News intelligence for banking analysts

Every morning, risk analysts at Société Générale hand-build the daily market digest — the briefing top management reads to grasp overnight moves. It means trawling dozens of sources, grouping related stories, picking representatives, and writing headlines. On average it eats three to four hours of senior-analyst time, every single day.

Three concrete gaps the analysts face:

This project built an NLP pipeline that does the grouping automatically. Each headline comes in, gets embedded as numbers, clustered by meaning, and labeled with its key names and topics. The goal: automate the grouping so analysts spend their time on judgment, not janitorial reading.

The pipeline runs in three stages: Embedding, Clustering, and Keyword Extraction. First we turn every headline into a vector, a string of numbers the software can read; then we group those vectors into clusters by meaning; finally we pull the defining keywords from each cluster.

Of the three stages, the embedding is where most of the difficulty lives — we need to make sure the headline's meaning survives during the conversion into a vector.

We evaluated six models, from the traditional to the state-of-the-art: TF-IDF, averaged Word2Vec, Doc2Vec, NVDM, the Universal Sentence Encoder (USE), and SentenceBERT.

The Transformer models clearly performed best by a clear margin. As they are able to group headlines by meaning, not shared words — a headline that says only "Biden" or "Beijing" still lands in the US-China cluster, even though neither word is "US-China." The keyword-based methods can't make those links, and dumped close to 80% of articles into one meaningless residual cluster. And among the transformer models, we have chosen BERT as it has performed the best result overall.

With every headline now a vector, semantically similar headlines sit close together, and clustering draws the boundaries between topics. We shortlisted three algorithms — Hierarchical Agglomerative Clustering (HAC), DBSCAN, and OPTICS — and judged them on two criteria: whether they work with cosine similarity (so a headline's meaning, not its length, drives the result) and how directly we could set the number of clusters.

HAC won on both counts. It takes cosine distance natively, and the number of clusters follows from a single linking threshold. It's also the approach trusted by industry practitioners like Bloomberg.

A single clustering pass is thorough but too granular: it splits the day's news into tens of small groups, still not so friendly for analysts. Recursive clustering fixes this by treating those groups as new inputs and clustering them a second time, so the fragments of a story merge back into one. It's the same two-stage design behind Bloomberg's NSTM news engine, and it's what turns a technically correct result into something an analyst can actually skim.

The final stage tries to understand what each cluster represents, by pulling out the names and topics its headlines mention most — its top named entities and top nouns — so the subject is obvious at a glance.

Named entity extraction runs on spaCy, an open-source NLP library we chose over its main rival NLTK for its faster, broader pretrained support; we use its largest English pipeline, en_core_web_lg. We extract the named entities found in the headlines and rank them by frequency to get the top named entities.

Named-entity recognition has a blind spot, though: brand-new terms it has never seen slip through — "dogecoin," absent from any 2020-era model. To cover that, we also pull each cluster's most frequent nouns as a second set of keywords.

We evaluated on three datasets: two public NLP benchmarks — 20 Newsgroups and the UCI News Aggregator — for scale and comparability, and a custom dataset we built to mirror the bank's real use case. The custom set is what makes the evaluation meaningful: it's drawn from the financial sources analysts actually read, organised under the topics they track, with a known answer key the clustering can be graded against.

The custom dataset is the one we built specifically for this project, to put the pipeline on the kind of news SG's analysts actually read. We gathered roughly 560 financial headlines from six outlets analysts trust — the Financial Times, Bloomberg, Reuters, the Wall Street Journal, the South China Morning Post and Risk.net, searching each for five themes: Brexit, Cryptocurrency, Electric Vehicles, Hong Kong and US-China relations.

Those five tags were chosen for financial relevance and for deliberate overlap — Hong Kong sits semantically next to US-China, and Tesla links electric vehicles to crypto — so the dataset tests whether the embeddings can tell genuinely confusable topics apart.

We are grateful for the opportunity to work on this project with the Société Générale Risk Management desk, it gave us the chance to take what we learned in the classroom and apply it to a live problem, bringing real value to real users.

From the implementation of the pipeline, we are pleased to see that our results achieved successful news headline clustering without any dataset-specific training, a meaningful outcome that speaks to the generalisability of the approach. We are glad to have delivered value to the SG team across three areas:

1. Serving as a first line of analysis through the daily flood of news headlines for SG's analysts
2. Enabling a systematic understanding of market behaviour and supporting the detection of anomalies
3. Providing a reusable foundation of insights for future NLP applications