Cue, an originate-provide info validation language
How did CUE reach about and what are its strategies.
Intro
CUE is an originate-provide info validation language and inference engine
with its roots in logic programming.
Though the language will not be a long-established-motive programming language,
it has many applications, similar to
info validation, info templating, configuration, querying,
code technology and even scripting.
The inference engine can even be former to validate
info in code or to embody it as portion of a code technology pipeline.
A key element that objects CUE other than its request languages
is that it merges kinds and values genuine into a single idea.
Whereas in most languages kinds and values are strictly obvious,
CUE orders them in a single hierarchy (a lattice, to be exact).
It’s a long way a extraordinarily extremely efficient idea that permits CUE to attain
many bask in issues.
It also simplifies issues.
For instance, there is not very such a thing as a necessity for generics and enums, sum kinds
and null coalescing are the final an identical element.
Applications
CUE’s invent ensures that combining CUE values in any
expose repeatedly presents the an identical end result
(it is associative, commutative and idempotent).
This makes CUE in particular smartly-suited for instances where CUE
constraints are blended from diversified sources:
-
Files validation: diversified departments or groups can each and each
define their have faith constraints to be aware to the an identical pickle of knowledge. -
Code extraction and technology: extract CUE definitions from
extra than one sources (Coast code, Protobuf), combine them genuine into a single
definition, and use that to generate definitions in a single other
structure (e.g. OpenAPI). -
Configuration: values can even be blended from diversified sources
with out one having to import the diversified.
The ordering of values also enables pickle containment evaluation of total
configurations.
The effect most validation systems are limited to checking whether a concrete
cost fits a schema, CUE can validate whether any occasion of
one schema is also an occasion of one other (is it backwards bask in minded?),
or compute a recent schema that represents all cases that match
two diversified schema.
Historical previous
Though it’s a extraordinarily diversified language, the roots of CUE lie in GCL,
the dominant configuration language in use at Google as of this writing.
It changed into initially designed to configure Borg, the predecessor of Kubernetes.
In actuality, the brand new idea changed into to use graph unification as former in CUE for GCL.
For sure one of the authors of GCL had in depth expertise with such systems and
skilled the advantage of being in a position to compute and reason with kinds for the
introduction of extremely efficient tooling.
The graph unification mannequin CUE is in accordance with
changed into in long-established use in computational linguistics in the in the intervening time and changed into
efficiently former to administer grammars and lexicons of over 100k lines of
declarative definitions.
These were successfully very trim
configurations of something as irregular and advanced as a human language.
A property of those systems were that the categories, or constraints, one
defines validate the guidelines whereas simultaneously decreasing boilerplate.
General, this come perceived to be extraordinarily smartly-suited
for cloud configuration.
Nevertheless, the early invent of GCL went for something extra purposeful that coincidentally
changed into also incompatible with the idea of graph unification.
This extra purposeful come proved inadequate, however it changed into already too leisurely to
switch to the earlier foreseen come.
As an different, an inheritance-based completely completely override mannequin changed into adopted.
Its complexity made the earlier foreseen tooling intractable
and they also below no instances materialized.
The an identical holds for the GCL offsprings that copied its mannequin.
CUE goes aid to the brand new idea of using a constraint-based completely completely come and
also makes an effort to embody lessons realized from 15 years of GCL utilization.
This also entails lessons realized from offsprings and diversified approaches to
configuration altogether.
Philosophy and methods
Sorts are Values
CUE would not distinguish between values and kinds.
It’s a long way a extremely efficient idea that permits CUE to define extremely-detailed
constraints, however it also simplifies issues considerably:
there is not very such a thing as a separate schema or info definition language to learn
and related language constructs similar to sum kinds, enums,
and even null coalescing collapse onto a single produce.
Below is an illustration of this idea.
On the left one can look a JSON object (in CUE syntax) with some properties
in regards to the city of Moscow.
The heart columns reveals a that it’s probably you’ll well well presumably also mediate of schema for any municipality.
On the ideal one sees a mixture between info and schema as is exemplary of CUE.
Files
moscow: {
establish: "Moscow"
pop: 11.92M
capital: valid
}Schema
municipality: {
establish: string
pop: int
capital: bool
}CUE
largeCapital: {
establish: string
pop: >5M
capital: valid
}In long-established, in CUE one begins with a principal definition of a form, describing
all that it’s probably you’ll well well presumably also mediate of cases.
One then narrows down these definitions, presumably by combining constraints
from diversified sources (departments, users), till a concrete info occasion
remains.
Push, not pull, constraints
CUE’s constraints act as info validators, however also double as
a mechanism to chop again boilerplate.
It’s a long way a extremely efficient come, however requires some diversified pondering.
With frail inheritance approaches one specifies the templates that
are to be inherited from at each and each level they desires to be former.
In CUE, as a change, one selects a pickle of nodes in the configuration to which
to be aware a template.
This selection can even be at a obvious level in the configuration altogether.
One other system to gaze this, a JSON configuration, voice, can even be
defined as a chain of direction-leaf values.
For instance,
{
"a": 3,
"b": {
"c": "foo"
}
}
would possibly be represented as
"a": 3
"b": "c": "foo"
The general info of the brand new JSON file is retained in this
illustration.
CUE generalizes this idea to the following pattern:
:
Every topic declaration in CUE defines a pickle of nodes to which to be aware
a particular constraint.
On chronicle of expose doesn’t topic, extra than one constraints can even be applied to the
same nodes, all of which must be aware simultaneously.
Such constraints would possibly well well even be in diversified files.
But they would well below no instances contradict each and each diversified:
if one declaration says a topic is 5, one other couldn’t override it to be 6.
Declaring a topic to be each and each >5 and <10 is precise, though.
This come is extra restricted than paunchy-blown inheritance;
it couldn't be that it's probably you'll well well presumably also mediate of to reuse unusual configurations.
On the diversified hand, additionally it's a extra extremely efficient boilerplate remover.
For instance, declare each and each job in a pickle desires to use a particular
template.
As an different of having to spell this out at each and each level,
one can divulge this individually in a one blanket verbalize.
So as a change of
jobs: {
foo: acmeMonitoring & { /... */ }
bar: acmeMonitoring & { /... */ }
baz: acmeMonitoring & { /... */ }
}
one can write
jobs: [string]: acmeMonitoring
jobs: {
foo: { /... */ }
bar: { /... */ }
baz: { /... */ }
}
There will not be a must repeat the reference to the monitoring template for
each and each job, as the primary already states that every person jobs must use acmeMonitoring.
Such requirements can even be specified across files.
This come not simplest reduces the boilerplate contained in acmeMonitoring
however also eliminates the repetitiveness of having to specify
this template for every and each job in jobs.
On the an identical time, this verbalize act as a form enforcement.
This twin characteristic is a key aspect of CUE and
typed characteristic building languages in long-established.
This come breaks down, of course, if the constraints in
acmeMonitoring are too stringent and jobs must override them.
To this extent, CUE presents mechanisms to enable defaults, decide-out, and
tender constraints.
Separate configuration from computation
There comes a time that one (seemingly) will need achieve complex
computations to generate some configuration info.
But simplicity of a configuration language can even be paramount when one hasty
desires to earn changes.
These are clearly conflicting pursuits.
CUE takes the stance that computation and configuration must quiet
be separated.
And CUE in point of fact makes this easy.
The records that desires to be computed can even be generated exterior of CUE
and keep in a file that is to be jumbled together.
The records also would possibly be generated in CUE’s scripting layer and automatically
injected in a configuration pipeline.
Both approaches depend upon CUE’s property that the expose in which this knowledge will get
added is beside the level.
Be precious at all scales
The usefulness of a language would possibly well well depend upon the scale of the mission.
Having too many diversified languages can keep a cognitive pressure on
developers, though, and migrating from one language to 1 other as
scaling requirements trade can even be very costly.
CUE targets to chop again these costs
by covering a myriad of knowledge- and configuration-related projects at all scales.
Puny scale
At small scales, decreasing boilerplate in configurations will not be necessarily
the ideal element to attain.
Even at a small scale, then yet again, repetition can even be error inclined.
For such instances, CUE can define schema to validate otherwise
typeless info files.
Medium scale
As soon the will arises to chop again boilerplate, the cue diagram can
wait on to automatically rewrite configurations.
Search for the Snappy and Soiled piece of the
Kubernetes tutorial
for an example using the import and orderly diagram.
Hundreds of lines can even be obliterated automatically using this come.
Immense scale
CUE’s underlying formalism changed into developed for trim-scale configuration.
Its import mannequin incorporates most absorbing practices for trim-scale engineering
and it is optimized for automation.
A key to here is evolved tooling.
The mathematical mannequin underlying CUE’s operations enables for
automation that is untractable for most diversified approaches.
CUE’s orderly divulge is an example of this.
Tooling
Automation is principal.
Within the intervening time, an valid chunk of code will get generated, analyzed, reformatted,
and heaps others by machines.
The CUE language, APIs, and tooling were designed to enable for
machine manipulation.
Aspects of this are:
- earn the language easy to scan and parse,
- restrictions on imports,
- enable any share of knowledge to be damage up across files and generated
from diversified sources, - define programs on the checklist stage,
- and of course its cost and form mannequin.
The expose independence also plays a key characteristic in this.
It enables combining constraints from varied sources with out having
to define any expose in which they are to be applied to earn
predictable outcomes.