Text Readability Prediction with Transformers

Highlights

• a complete PyTorch / Hugging Face text regression pipeline
• BERT, RoBERTa and related transformers with custom pooling heads
• an interactive app that scores the reading complexity of any custom text

Tags: natural language processing, deep learning, web app

Summary

Estimating how difficult a text is to read matters to teachers: students develop reading skills fastest when passages are challenging but not overwhelming. Classic readability formulas rely on weak proxies such as sentence length and syllable counts, which leaves a lot of accuracy on the table.

The project replaces those heuristics with an ensemble of eight transformer models. Each model uses a regression head on top of a concatenation of several hidden layers, which captures both surface-level and semantic signals. Training relies on text augmentation – sentence-order shuffling, back-translation and target noise – to make the most of roughly 2,800 labelled passages from the CommonLit Readability Prize data set.

Two of the trained models power a small Streamlit app: paste a paragraph and the app returns a readability score together with example texts for comparison. The blog post on text readability walks through the pipeline and the app code.

Image-to-Text Translation of Molecules

Highlights

• a CNN-LSTM encoder-decoder that turns structure drawings into chemical formula strings
• a PyTorch image captioning pipeline that runs on both GPUs and TPUs
• an ensemble of encoder-decoder models evaluated with the Levenshtein distance

Tags: computer vision, natural language processing, deep learning

Summary

Chemists usually draw molecules as structural diagrams. Decades of scanned publications and medical documents therefore contain images that are not annotated with machine-readable formulas, and converting them reliably requires time-consuming expert work. Automated recognition of optical chemical structures could speed up research and development in the field.

The project frames the task as image captioning: translating an unlabelled chemical image into its InChI text string. An EfficientNet encoder extracts features from each drawing, and a multi-layer LSTM decoder generates the formula token by token. Training uses the Bristol-Myers Squibb Molecular Translation data set with more than four million chemical images, so an efficient data pipeline matters as much as the architecture.

Predictions are scored with the Levenshtein distance between the generated and the true string – the number of character edits needed to fix a prediction. Combining several encoder-decoder models into an ensemble lowers that distance compared with any single model.

Fair Machine Learning in Credit Scoring

Highlights

• a benchmark of eight fairness processors on seven credit scoring data sets
• a profit-fairness trade-off analysis that puts a price on fairness
• practical guidance on choosing a fairness criterion for scorecards

Tags: tabular data, fairness, profit maximization

Summary

Lenders use scorecards to approve or reject loan applications, which makes algorithmic discrimination a real concern. The project looks at fairness from three angles:

• which statistical fairness criteria – independence, separation or sufficiency – make sense for credit decisions
• where fairness goals can enter the modelling pipeline: pre-processing the data, constraining the learner, or post-processing the scores
• how much profit a lender gives up when a fairness constraint is enforced on real-world data

The experiments show that several criteria can be approximately satisfied at the same time, that separation is a sensible default for scorecards, and that in-processing methods tend to offer the best balance between profit and fairness. The tutorial on algorithmic fairness in R shows how to compute the underlying metrics.