A Fistful of States: More Order Machine Patterns in Rust

Ana introduces a likelihood of necessities for a staunch remark machine
implementation, and achieves them with concise and with out negate interpretable code.
In explicit, these requirements (some based on the definition of a remark
machine, and some on ergonomics) were a high precedence for us:

• One remark at a time.
• Ability for shared remark.
• Top possible explicitly defined transitions ought to be permitted.
• Any error messages ought to be easy to trace.
• As many errors as possible ought to be known at assemble-time.

Within the subsequent fragment I would possibly declare about some extra requirements that we
launched and how these impacted the answer. In explicit, we relaxed some
of Ana’s targets in alternate for greater flexibility, whereas gratifying those
listed above.

Tribulation

We were off to a pleasant launch, but it completely used to be time to bear in solutions how we desire
downstream developers to procure interplay with our fresh remark machine API. In
explicit, whereas the Kubelets we are aware of all note roughly the the same
Pod lifecycle, we wished developers to be ready to place in power arbitrary remark
machines for their Kubelet. Shall we embrace, some workloads or architectures can also
must procure extra provisioning states for infrastructure or details, or
to introduce put up-flee states for like minded rubbish series of sources.
Additionally, it felt indulge in an anti-sample to procure a mum or dad capacity (main in
the instance above) which defines the good judgment for progressing thru the states,
as this felt indulge in having two sources of truth and used to be now not one thing we can even
put in power on behalf of our downstream developers for arbitrary remark machines.
Ana had talked about easy solutions to retain the remark machine in a mum or dad structure the employ of an
enum, but it completely felt clunky to introduce tidy match statements which can also
introduce runtime errors.

We knew that to permit arbitrary remark machines we would wish a Order trait to
mark kinds as legit states. We felt that it would possibly possibly possibly well be possible for this trait to
procure a next() capacity which runs the remark and then returns the subsequent Order
to transition to, and we wished our code to be ready to merely call next()
time and all but again to power the machine to completion. This sample, we soon found,
launched a likelihood of challenges.

/// Tough pseudocode of our idea.

trait Order {
    /// Lift out work for this remark and return next remark.
    async fn next(self) -> impl Order;
}

fn drive_state_machine(mut remark: impl Order) {
    loop {
        remark = remark.next().handle up for;
    }
}

What does next() return?

Internal our loop, we are time and all but again overwriting a local variable with
different kinds that every person put in power Order. Without our mum or dad capacity, or
wrapper enum, there used to be now not a easy capacity for Rust to know easy solutions to
store these objects on the stack. Pointless to reveal, the Rust compiler used to be
displeased. We spent some time pair programming with the Rust Playground on
solutions to this, and settled on the employ of trait objects (Field), which
moves the article itself to the heap.

The usage of the heap violates one of Ana’s fresh targets, and we discovered that
the usage of trait objects introduces quite so a lot of (obligatory) barriers on the
trait itself, at the side of that it disallows generic solutions and referencing Self in return kinds.
This doesn’t end us from achieving our targets, but is
limiting and reduces efficiency attributable to the usage of dynamic dispatch. We will
proceed to stumble on stack-based solutions.

With the usage of trait objects, there are handiest two alternate concepts for the return form
of next(), which we captured with an enum:

pub enum Transition {
    /// Transition to a brand fresh remark.
    Next(Field),
    /// Stop execution of remark machine with end result.
    Entire(End result<()>)
}

A Little bit of Folly

We temporarily explored one option for warding off boxed traits: in living of iterating
on a trait object, we can even clarify a recursive operate, which accepts a remark,
runs its handler, and then calls itself with the subsequent remark. This would living
all of our remark objects on the stack by rising it with every recursive call,
and felt indulge in a extraordinarily entertaining solution which leveraged Rust’s fresh
impl Trait characteristic, as successfully as the async_recursion crate. Pointless to claim we
realized that this used to be now not acceptable because Pods time and all but again obtain caught in
loops (such as image pull / image pull backoff), which would possibly possibly develop the stack
with out limit. Every other drawback used to be that with concrete generic kinds,
Transition had to procure an enum variant for every remark that will be
transitioned to in a given remark handler, making it impractical to bolster more
than a pair of outgoing edges in the remark graph.

/// Represents end result of remark execution and which remark to transition to.
pub enum Transition {
    /// Reach to next remark.
    Reach(S),
    /// Transition to error remark.
    Error(E),
    /// That is a terminal node of the remark graph.
    Entire(End result<()>),
}

#[async_recursion::async_recursion]
/// Recursively bear in solutions remark machine until a remark returns Entire.
pub async fn run_to_completion(remark: impl Order) -> End result<()> {
    let transition = { remark.next().handle up for? };

    match transition {
        Transition::Reach(s) => run_to_completion(s).handle up for,
        Transition::Error(s) => run_to_completion(s).handle up for,
        Transition::Entire(end result) => end result,
    }
}

Kubernetes-explicit Habits

A more effective process used to be at the side of habits to the Order trait to bolster our wants.
This incorporated at the side of context to the next() capacity, such as the Pod manifest
and a generic form, PodState, to wait on as shared details between remark handlers.
This details is now not shared between Pods, so remark handlers from different Pods
can develop concurrently. For any remark shared between Pods, we chose to
trudge away it to developers to place in power concurrency controls, which must tranquil develop it
mighty more glaring to them when one Pod’s execution is blocking one other’s.

Next, we added a second capacity that would possibly possibly well be known as upon when coming into a remark,
which must tranquil develop a JSON patch for the Pod role connected with that
remark. This update is then sent to the Kubernetes retain a watch on plane by our API, and
this is ideally the one living in the code where Pod role patches are
utilized.

With all this in solutions, we discontinue up with a trait and a operate to iteratively
power it to completion:

#[async_trait::async_trait]
/// A trait representing a node in the remark graph.
pub trait Order: Sync + Ship + 'static + std::fmt::Debug {
    /// Bustle remark handler and return next remark or full.
    async fn next(
        // We *areready to make employ of `Self` here, allowing us to pass the article, 
        // and preventing reuse.
        self: Field,
        // Pod remark shared between remark handlers.
        pod_state: &mut PodState,
        // Pod manifest.
        pod: &Pod, 
    ) -> anyhow::End result>;

    /// Returns JSON role patch to employ when coming into this remark.
    async fn json_status(
        &self,
        pod_state: &mut PodState,
        pod: &Pod,
    ) -> anyhow::End result;
}

/// Iteratively bear in solutions remark machine until it returns Entire.
pub async fn run_to_completion(
    consumer: &kube::Client,
    remark: impl Order,
    pod_state: &mut PodState,
    pod: &Pod,
) -> anyhow::End result<()> {
    let api: Api = Api::namespaced(consumer.clone(), pod.namespace());

    let mut remark: Field> = Field::fresh(remark);

    loop {
        let patch = remark.json_status(pod_state, &pod).handle up for?;
        let details = serde_json::to_vec(&patch)?;
        api.patch_status(&pod.title(), &PatchParams::default(), details)
            .handle up for?;

        let transition = { remark.next(pod_state, &pod).handle up for? };

        remark = match transition {
            Transition::Next(s) => {
                s.remark
            }
            Transition::Entire(end result) => {
                destroy end result;
            }
        };
    }
}

The manner to place in power edges

The final mission that we tackled used to be easy solutions to place in power constraints on which remark
transitions are legit. After we moved to the employ of trait objects, we successfully
made our remark machine a fully linked graph. When objects are boxed and
returned as trait objects, their form is successfully misplaced, moreover the trait
they put in power. This meant that remark handlers can also return one thing else that’s
Order, and it would assemble. We wished to search out a mode to explicitly clarify
legit remark transitions, as proven in Ana’s solution, and transparently
put in power this with the API we had developed.

This resulted in a devious idea. Beneath the guise of providing a pleasant
static capacity to handle Boxing for the user, we would add a where clause to
develop optimistic a directed edge trait is utilized between the 2 Orders.
Within the end, the remark itself would possibly possibly well be wrapped in a struct with a non-public field,
which would possibly possibly end manual development of Transition::Next with out the employ of
this static capacity (exterior of the kublet crate, now not much less than).

/// Implementor can transition to remark `S`.
pub trait TransitionTo {}

pub struct StateHolder { 
    // This field is non-public.
    remark: Field 
}

impl Transition {
    pub fn next(
        _t: ThisState, 
        n: NextState
    ) -> Transition where ThisState: TransitionTo,
    {
        Transition::Next(StateHolder { remark: Field::fresh(s) })
    }
}

Transition::next(self, NextStateObject)

struct PodState;

struct Starting;

#[async_trait::async_trait]
impl Order for Starting {
    async fn next(
        self: Field,
        pod_state: &mut PodState,
        pod: &Pod,
    ) -> anyhow::End result> {
        // TODO: launch workload 
        Good ample(Transition::next(Running))
    }

    async fn json_status(
        &self,
        pod_state: &mut PodState,
        pod: &Pod,
    ) -> anyhow::End result {
        Good ample(serde_json::json!(null))
    }
}

struct Running;

#[async_trait::async_trait]
impl Order for Running {
    async fn next(
        self: Field,
        pod_state: &mut PodState,
        pod: &Pod,
    ) -> anyhow::End result> {
        // Bustle forever
        loop {
            tokio::time::delay_for(std::time::Length::from_secs(10)).handle up for;
        }
    }

    async fn json_status(
        &self,
        pod_state: &mut PodState,
        pod: &Pod,
    ) -> anyhow::End result {
        Good ample(serde_json::json!(null))
    }
}

impl TransitionTo for Starting {}

struct Terminated;

#[async_trait::async_trait]
impl Order for Terminated {
    async fn next(
        self: Field,
        pod_state: &mut PodState,
        pod: &Pod,
    ) -> anyhow::End result> {
        // TODO: interrupt workload
        Good ample(Transition::Entire(Good ample(())))
    }

    async fn json_status(
        &self,
        pod_state: &mut PodState,
        pod: &Pod,
    ) -> anyhow::End result {
        Good ample(serde_json::json!(null))
    }
}

struct ExampleProvider;

impl Provider for ExampleProvider {
    form PodState = PodState;

    form InitialState = Starting;

    form TerminatedState = Terminated;

    /// Employ this hook to inject remark shared between Pods into `PodState` 
    /// earlier than it is handed to the remark machine.
    async fn initialize_pod_state(
        &self, 
        pod: &Pod
    ) -> anyhow::End result {
        Good ample(PodState)
    }
}