D Curious Tech

Tag: software-engineering

  • Sustainable AI Engineering

    and no, not the carbon kind

    Whether the AI system you shipped last quarter will still be running next year without being rebuilt from scratch.

    When people write about sustainability and AI, they usually mean electricity. Datacentre load, water consumption, the emissions cost of a training run.

    That’s a real conversation. It isn’t this one.

    I want to talk about a different kind of sustainability: whether the AI system your team shipped last quarter will still be running next year without being rebuilt from scratch.

    Because most of them won’t be.

    Software decays. AI systems expire.

    Traditional software ages gracefully. Dependencies drift, documentation goes stale, the odd library hits end-of-life — but the thing keeps running. Decay is slow, visible and negotiable.

    AI systems don’t work like that. They fail discontinuously.

    A model gets deprecated and your application stops working on a date set by someone else’s roadmap. A provider updates a checkpoint and your carefully tuned prompt quietly starts producing different output — not worse in any way that throws an error, just different enough to matter.

    And occasionally the model simply disappears. In June, three days after Anthropic released Claude Fable 5, a US export control directive forced it offline for every customer worldwide. It came back on 1 July once the controls were lifted. Nineteen days is survivable. The point is that nobody in the dependency chain chose it — not the customers, not the vendor — and almost no architecture review had that failure mode on the list.

    The uncomfortable truth is that most AI systems being built right now are built against a specific model, a specific prompt and a specific moment. They aren’t built to last. They’re built to demo.

    Sustainable AI engineering is the discipline of building systems that survive the thing underneath them changing.

    Here’s what that actually requires.

    1. Your foundation has an expiry date

    Cloud infrastructure gave us a decade of stability. An EC2 instance you provisioned in 2016 behaves the same way today. That stability quietly shaped how we build.

    Foundation models offer nothing of the sort. The useful life of a frontier model is somewhere between six and eighteen months, and the decision to retire it isn’t yours.

    This means model substitution has to be a design assumption, not a migration project. If swapping the model behind a feature requires a sprint of prompt rewriting and manual spot-checking, you don’t have a system. You have a dependency you can’t move.

    The June suspension was the dramatic version of this problem. The mundane version — a deprecation notice with ninety days on it — will hit far more teams, far more often, and get far less attention.

    2. Evals are the new regression suite

    We have thirty years of collective muscle memory around testing deterministic software. Write the test, get a pass or a fail, gate the deploy.

    None of that transfers cleanly. The same input produces different outputs. Correctness is often a judgement rather than an assertion. So a lot of teams have quietly stopped testing and started eyeballing.

    Which means they have no way to answer the only question that matters when something changes: did that break anything?

    The tempting shortcut is to reach for a similarity score and call it a test. That doesn’t hold either — I’ve written before about why BERTScore and cosine similarity aren’t enough, and the short version is that they will happily pass an answer that is fluent, plausible and wrong.

    Without a real evaluation suite, every model upgrade is a leap of faith, every prompt tweak is unverified, and every regression is discovered by a customer. Eval debt behaves like technical debt, except it compounds faster and you can’t see the balance.

    The teams building sustainably treat evals as infrastructure — versioned, automated, run in CI, and owned by someone.

    3. Prompts and context are production code

    In a lot of organisations, the prompt that drives a customer-facing feature lives in a Confluence page, a notebook cell, or a string literal someone edited on a Friday afternoon.

    This is config on somebody’s laptop, wearing a new hat. We solved this problem once already and appear determined to relearn it.

    Prompts, retrieval configurations, tool definitions and context assembly logic are the highest-leverage, highest-volatility parts of an AI system. They change behaviour more than the code around them. They deserve version control, code review, change history and a rollback path.

    This is also the part that makes incidents survivable. As I covered in the AWS governance playbook, every interaction should be traceable back to the system prompt version, model version, guardrail version and knowledge source version that produced it. Without that, an investigation becomes archaeology.

    If you can’t answer “what changed, when, and who approved it” for a prompt, you cannot operate that system responsibly — and you certainly can’t debug it six months from now.

    4. Every AI feature needs a failure mode

    Ask most teams what their AI feature does when the model is rate-limited, degraded, timing out or unavailable, and the honest answer is: it fails.

    That’s not a resiliency posture. That’s a single point of failure with a language model attached.

    Sustainable systems have somewhere to go when the primary path is unavailable:

    • A fallback model, ideally from a different provider
    • A deterministic path that’s worse but functional
    • Graceful degradation — narrower scope, cached results, reduced capability
    • A clean handoff to a human
    • Circuit breakers, so a degraded model doesn’t quietly poison a thousand downstream decisions

    We wouldn’t ship a payments integration with no retry logic and no fallback. We’re shipping AI features that way constantly, and the reason is that the failure modes are new enough that nobody’s asked the question yet.

    5. Cost is a runtime property, not a line item

    Traditional software has broadly fixed unit economics. AI systems don’t. Cost per interaction is a live variable that moves when the model changes, when the context window grows, when an agent decides to take twelve steps instead of three.

    Plenty of AI systems get switched off despite working perfectly. They just cost more per transaction than the transaction is worth, and nobody modelled that until the invoice arrived.

    If cost isn’t observable per feature, per user and per interaction — in something closer to real time than a monthly bill — it isn’t controllable. Token anomaly detection is the early warning; connecting that spend to a defensible outcome is what gets the system through its first budget review.

    6. A system nobody owns isn’t sustainable

    The barrier to building something impressive with AI has collapsed. That’s genuinely good. It also means organisations are accumulating agents, assistants and pipelines built quickly by people who have since moved on.

    Who maintains the agent someone built in a weekend? Who gets paged when it starts making bad calls? Who decides whether it still needs to exist?

    Ownership is the question that the agent control plane keeps running into from every direction — who owns the tools, the policies, the prompts, the incident response. It’s the least technical item on this list and probably the one that kills the most systems. Engineering rigour is irrelevant if nobody owns the outcome.

    The throughline

    None of this is exotic. It’s mostly the operational discipline we already apply to everything else, redirected at a substrate that changes far faster than we’re used to.

    But notice what these six things have in common: model abstraction, evaluation infrastructure, prompt management, fallback routing, cost observability, ownership. Almost none of them are application concerns. They’re platform concerns — the same argument for a dedicated AI platform team, and the same argument I made about the AI operating model, arriving from a different direction.

    Asking every product team to independently invent resiliency is how you end up with none.

    The organisations pulling ahead over the next year won’t be the ones that shipped AI features fastest. They’ll be the ones that don’t have to rebuild them.

    Is your AI stack built to survive the model underneath it changing — or would a deprecation notice mean starting over?

  • Java Streams

    Today we will look at Streams in Java

    An example of Java Streams to print the even numbers is as follows

    import java.util.Arrays;
import java.util.List;

public class StreamsSamples {
    public static void main(String[] args) {
        List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5);
        numbers.stream()
                .filter(a -> a % 2 == 0)
                .forEach(a -> System.out.println("Even number: " + a));
    }
}

    Java Streams come in two flavours .stream() and .parallelStream(). Below is a quick comparison of the two

    FeatureStreamParallelStream
    ExecutionSequential (one element at a time)Parallel (multiple elements simultaneously)
    ThreadingSingle-threadedMulti-threaded (uses ForkJoinPool)
    PerformanceMay be slower for large datasetsCan be faster for large datasets with CPU cores
    Order PreservationMaintains encounter orderMay not preserve order (unless explicitly stated)
    Use CaseSmall to medium datasets, order-sensitive opsLarge datasets, CPU-intensive operations
    DeterminismMore predictable and deterministicMay have non-deterministic results
    Side EffectsEasier to manageHarder to control due to concurrent execution
    OverheadLowHigher due to thread management overhead
    Custom Thread PoolNot requiredUses common ForkJoinPool (customization is tricky)
    Exampleslist.stream()list.parallelStream()

    As highlighted in the above table, ParallelStream is not useful when the dataset count is very small to medium. This adds additional overhead of multiple threads creation and their lifecycle management.

    Lets look at the below example of identifying a prime number in about 1000 numbers

    package com.dcurioustech.streams;

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class StreamsSamples {
    public static void main(String[] args) {
        System.out.println("================================");
        // Inefficient use of parallel streams
        List<Integer> largeNumbers = new java.util.Random().ints(1_000, 1, 1000).boxed().collect(Collectors.toList());
        System.out.println("Sample count:" + largeNumbers.size());

        // Using sequential streams
        long startTime = System.nanoTime();
        largeNumbers.stream().filter(StreamsSamples::isPrime).count();
        long endTime = System.nanoTime();
        float sequentialTime = endTime - startTime;
        System.out.println("Sequential stream time (milli seconds): " + (sequentialTime)/1_000_000);

        // Using parallel streams
        startTime = System.nanoTime();
        largeNumbers.parallelStream().filter(StreamsSamples::isPrime).count();
        endTime = System.nanoTime();
        float parallelTime = endTime - startTime;
        System.out.println("Parallel stream time (milli seconds): " + (parallelTime)/1_000_000);
        System.out.println("Speedup: " + sequentialTime/parallelTime);

    }

    // Intentionally inefficient CPU intensive method
    public static boolean isPrime(int number) {
        if (number <= 1) {
            return false;
        }
        for (int i = 2; i < number; i++) {
            if (number % i == 0) {
                return false;
            }
        }
        return true;
    }
}

    Output as below:
    ================================
    Sample count:1000
    Sequential stream time (milli seconds): 1.867237
    Parallel stream time (milli seconds): 5.67832
    Speedup: 0.32883617

    As can be seen the ParallelStream time is more than the Sequential stream. This is due to the overhead of thread life cycle management.

    Lets now look at the example of about 10 million sized sample

    package com.dcurioustech.streams;

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class StreamsSamples {
    public static void main(String[] args) {
        System.out.println("================================");
        // Efficient use of sequential streams
        List<Integer> largeNumbers = new java.util.Random().ints(10_000_000, 1, 1000).boxed().collect(Collectors.toList());
        System.out.println("Sample count:" + largeNumbers.size());

        // Using sequential streams
        long startTime = System.nanoTime();
        largeNumbers.stream().filter(StreamsSamples::isPrime).count();
        long endTime = System.nanoTime();
        long sequentialTime = endTime - startTime;
        System.out.println("Sequential stream time (milli seconds): " + (sequentialTime)/1_000_000);

        // Using parallel streams
        startTime = System.nanoTime();
        largeNumbers.parallelStream().filter(StreamsSamples::isPrime).count();
        endTime = System.nanoTime();
        long parallelTime = endTime - startTime;
        System.out.println("Parallel stream time (milli seconds): " + (parallelTime)/1_000_000);
        System.out.println("Speedup: " + sequentialTime/parallelTime);
    }

    // Intentionally inefficient CPU intensive method
    public static boolean isPrime(int number) {
        if (number <= 1) {
            return false;
        }
        for (int i = 2; i < number; i++) {
            if (number % i == 0) {
                return false;
            }
        }
        return true;
    }
}

    Output as below

    ================================
    Sample count:10000000
    Sequential stream time (milli seconds): 1978.1862
    Parallel stream time (milli seconds): 589.46625
    Speedup: 3.3558939

    As seen from the results, the performance with the use of parallel streams is 3.35 times faster

    Summary

    Stick to Sequential streams when
    > Sample size is small to medium
    > Order of the execution matters in the stream

    Use Parallel streams when
    > Sample size is large
    > Order of execution doesn’t matter

    Java streams are powerful and can improve the performance significantly for certain operations and large datasets, while also improving code readability over normal iterative constructs.

    You can refer to the code in here

More posts