Ben's Engineering Blog

A JSON parser in less than 250 lines-of-code

2026-01-28T00:00:00Z

Over the winter break I spent some time working on my JSON parser I wrote in late 2023. I had seen that with additions I made over the year, it had become a little bloated and had gone above an initially recorded statistic for line-of-code.

With some simplification to the existing logic I managed to get it to around 250 lines-of-code of Rust counted from the fn keyword to the last } brace of the main parsing function.

This is a quick post outlining some of the tricks and choices made when building this JSON parser.

The library is open-source and on crates.io but the latest code is under this PR and yet to merge at the time of writing.

With such a short implementation, I can paste it in its full glory and explain it later!

Some definitions elided, full code here

pub fn parse_with_options<'a, T>(
    on: &'a str,
    options: &ParseOptions,
    mut cb: impl for<'b> FnMut(&'b [JSONKey<'a>], RootJSONValue<'a>) -> Option<T>,
) -> Result<(usize, Option<T>), JSONParseError> {
    fn find_non_escaped_quoted(on: &str) -> Option<usize> {
        on.match_indices('"').find_map(|(idx, _)| {
            let rev = on[..idx].bytes();
            let last = rev.rev().take_while(|chr: &u8| *chr == b'\\').count();
            (last % 2 == 0).then_some(idx)
        })
    }

    fn parse_comment(on: &str) -> Option<(&str, usize)> {
        if let Some(rest) = on.strip_prefix('#') {
            let offset = rest.find('\n').unwrap_or(rest.len());
            Some((&on[..offset], offset + 1))
        } else if let Some(rest) = on.strip_prefix("//") {
            let offset = rest.find('\n').unwrap_or(rest.len());
            Some((&on[..offset], offset + 2))
        } else if let Some(rest) = on.strip_prefix("/*") {
            let offset = rest.find("*/")?;
            Some((&rest[..offset], offset + 4))
        } else {
            None
        }
    }

    let (mut idx, mut key_chain, mut in_object): (usize, Vec<JSONKey<'_>>, bool) =
        Default::default();
    let (bytes, tls) = (on.as_bytes(), options.top_level_separator);

    if tls.is_some() {
        key_chain.push(JSONKey::Index(0));
    }

    macro_rules! emit {
        ($item:expr) => {
            // TODO pass idx
            let res = cb(&key_chain, $item);
            if res.is_some() {
                return Ok((idx, res));
            }
        };
    }

    macro_rules! return_err {
        ($reason:ident) => {
            return Err(JSONParseError {
                at: idx,
                reason: JSONParseErrorReason::$reason,
            });
        };
    }

    macro_rules! skip_whitespace_find_comments {
        () => {
            while let Some(byte) = bytes.get(idx) {
                if let b' ' | b'\t' | b'\r' | b'\n' = byte {
                    idx += 1;
                } else if let b'/' | b'#' = byte {
                    let Some((comment, offset)) = parse_comment(&on[idx..]) else {
                        return_err!(InvalidComment);
                    };
                    idx += offset;
                    if options.yield_comments {
                        emit!(RootJSONValue::Comment(comment));
                    }
                } else {
                    break;
                }
            }
        };
    }

    skip_whitespace_find_comments!();

    while idx < bytes.len() {
        if in_object {
            skip_whitespace_find_comments!();
            if let Some(b'"') = bytes.get(idx) {
                let rest = &on[1..][idx..];
                let Some(offset) = find_non_escaped_quoted(rest) else {
                    return_err!(ExpectedQuote);
                };
                key_chain.push(JSONKey::Slice(&rest[..offset]));
                idx += offset + 2;
                skip_whitespace_find_comments!();
                if on.as_bytes().get(idx).copied().unwrap_or_default() != b':' {
                    // TODO partial could find next ':'?
                    return_err!(ExpectedColon);
                }
                idx += 1;
            } else {
                return_err!(ExpectedKey);
            }
        }

        skip_whitespace_find_comments!();

        match bytes.get(idx).copied() {
            Some(b'{') => {
                idx += 1;
                // little hack
                skip_whitespace_find_comments!();
                if let Some(b'}') = bytes.get(idx) {
                    idx += 1;
                    emit!(RootJSONValue::EmptyObject);
                } else {
                    in_object = true;
                    continue;
                }
            }
            Some(b'[') => {
                idx += 1;
                key_chain.push(JSONKey::Index(0));
                in_object = false;
                continue;
            }
            Some(b']') => {
                idx += 1;
                match key_chain.pop() {
                    Some(JSONKey::Index(0)) => {
                        emit!(RootJSONValue::EmptyArray);
                    }
                    Some(JSONKey::Index(_)) if options.allow_trailing_commas => {}
                    _ => {
                        return_err!(ExpectedEndOfValue);
                    }
                }
                in_object = matches!(key_chain.last(), Some(JSONKey::Slice(_)));
            }
            Some(b'"') => {
                let rest = &on[idx..][1..];
                let Some(offset) = find_non_escaped_quoted(rest) else {
                    return_err!(ExpectedEndOfValue);
                };
                idx += offset + 2;
                emit!(RootJSONValue::String(JSONString(&rest[..offset])));
            }
            Some(b'0'..=b'9' | b'-') => {
                /// can include bad values
                fn find_non_number(chr: char) -> bool {
                    !matches!(chr, '0'..='9' | '.' | 'e' | 'E' | '+' | '-')
                }

                let rest = &on[idx..];
                let Some(offset) = rest.find(find_non_number) else {
                    return_err!(ExpectedEndOfValue);
                };
                idx += offset;
                emit!(RootJSONValue::Number(JSONNumber(&rest[..offset])));
            }
            Some(b't') if on[idx..].starts_with("true") => {
                idx += 4;
                emit!(RootJSONValue::Boolean(true));
            }
            Some(b'f') if on[idx..].starts_with("false") => {
                idx += 5;
                emit!(RootJSONValue::Boolean(false));
            }
            Some(b'n') if on[idx..].starts_with("null") => {
                idx += 4;
                emit!(RootJSONValue::Null);
            }
            Some(b @ (b',' | b'}')) if options.partial_syntax => {
                emit!(RootJSONValue::Empty);
                idx += 1;
                if in_object {
                    let _ = key_chain.pop();
                }
                if b == b',' {
                    continue;
                }
            }
            _ => {
                return_err!(ExpectedValue);
            }
        }

        while let Some(byte) = bytes.get(idx) {
            if key_chain.is_empty() {
                return Ok((idx, None));
            }

            if tls.is_some_and(|c: char| on[idx..].starts_with(c)) {
                idx += 1;
                if let [JSONKey::Index(ref mut idx)] = key_chain.as_mut_slice() {
                    *idx += 1;
                    break;
                }
            } else if let b' ' | b'\t' | b'\r' | b'\n' = byte {
                idx += 1;
            } else if let b'/' | b'#' = byte {
                let Some((comment, offset)) = parse_comment(&on[idx..]) else {
                    return_err!(InvalidComment);
                };
                idx += offset;
                if options.yield_comments {
                    emit!(RootJSONValue::Comment(comment));
                }
            } else {
                let new_byte = if *byte == b',' {
                    idx += 1;
                    if let Some(JSONKey::Index(ref mut idx)) = key_chain.last_mut() {
                        *idx += 1;
                    } else {
                        key_chain.pop();
                    }
                    if !options.allow_trailing_commas {
                        break;
                    }
                    skip_whitespace_find_comments!();
                    let Some(b @ (b'}' | b']')) = bytes.get(idx) else {
                        break;
                    };
                    *b
                } else {
                    *byte
                };
                match new_byte {
                    b'}' if *byte == b',' || in_object => {}
                    b']' if matches!(key_chain.last(), Some(JSONKey::Index(_))) => {}
                    _ => {
                        return_err!(ExpectedEndOfValue);
                    }
                }
                idx += 1;
                key_chain.pop();
                in_object = matches!(key_chain.last(), Some(JSONKey::Slice(_)));
            }
        }
    }

    let tl = tls.is_some_and(|_| matches!(key_chain.as_slice(), &[JSONKey::Index(_)]));

    if !(key_chain.is_empty() || tl) {
        return_err!(ExpectedBracket);
    }

    Ok((on.len(), None))
}
pub fn parse_with_options<'a, T>(
    on: &'a str,
    options: &ParseOptions,
    mut cb: impl for<'b> FnMut(&'b [JSONKey<'a>], RootJSONValue<'a>) -> Option<T>,
) -> Result<(usize, Option<T>), JSONParseError> {
    fn find_non_escaped_quoted(on: &str) -> Option<usize> {
        on.match_indices('"').find_map(|(idx, _)| {
            let rev = on[..idx].bytes();
            let last = rev.rev().take_while(|chr: &u8| *chr == b'\\').count();
            (last % 2 == 0).then_some(idx)
        })
    }

    fn parse_comment(on: &str) -> Option<(&str, usize)> {
        if let Some(rest) = on.strip_prefix('#') {
            let offset = rest.find('\n').unwrap_or(rest.len());
            Some((&on[..offset], offset + 1))
        } else if let Some(rest) = on.strip_prefix("//") {
            let offset = rest.find('\n').unwrap_or(rest.len());
            Some((&on[..offset], offset + 2))
        } else if let Some(rest) = on.strip_prefix("/*") {
            let offset = rest.find("*/")?;
            Some((&rest[..offset], offset + 4))
        } else {
            None
        }
    }

    let (mut idx, mut key_chain, mut in_object): (usize, Vec<JSONKey<'_>>, bool) =
        Default::default();
    let (bytes, tls) = (on.as_bytes(), options.top_level_separator);

    if tls.is_some() {
        key_chain.push(JSONKey::Index(0));
    }

    macro_rules! emit {
        ($item:expr) => {
            // TODO pass idx
            let res = cb(&key_chain, $item);
            if res.is_some() {
                return Ok((idx, res));
            }
        };
    }

    macro_rules! return_err {
        ($reason:ident) => {
            return Err(JSONParseError {
                at: idx,
                reason: JSONParseErrorReason::$reason,
            });
        };
    }

    macro_rules! skip_whitespace_find_comments {
        () => {
            while let Some(byte) = bytes.get(idx) {
                if let b' ' | b'\t' | b'\r' | b'\n' = byte {
                    idx += 1;
                } else if let b'/' | b'#' = byte {
                    let Some((comment, offset)) = parse_comment(&on[idx..]) else {
                        return_err!(InvalidComment);
                    };
                    idx += offset;
                    if options.yield_comments {
                        emit!(RootJSONValue::Comment(comment));
                    }
                } else {
                    break;
                }
            }
        };
    }

    skip_whitespace_find_comments!();

    while idx < bytes.len() {
        if in_object {
            skip_whitespace_find_comments!();
            if let Some(b'"') = bytes.get(idx) {
                let rest = &on[1..][idx..];
                let Some(offset) = find_non_escaped_quoted(rest) else {
                    return_err!(ExpectedQuote);
                };
                key_chain.push(JSONKey::Slice(&rest[..offset]));
                idx += offset + 2;
                skip_whitespace_find_comments!();
                if on.as_bytes().get(idx).copied().unwrap_or_default() != b':' {
                    // TODO partial could find next ':'?
                    return_err!(ExpectedColon);
                }
                idx += 1;
            } else {
                return_err!(ExpectedKey);
            }
        }

        skip_whitespace_find_comments!();

        match bytes.get(idx).copied() {
            Some(b'{') => {
                idx += 1;
                // little hack
                skip_whitespace_find_comments!();
                if let Some(b'}') = bytes.get(idx) {
                    idx += 1;
                    emit!(RootJSONValue::EmptyObject);
                } else {
                    in_object = true;
                    continue;
                }
            }
            Some(b'[') => {
                idx += 1;
                key_chain.push(JSONKey::Index(0));
                in_object = false;
                continue;
            }
            Some(b']') => {
                idx += 1;
                match key_chain.pop() {
                    Some(JSONKey::Index(0)) => {
                        emit!(RootJSONValue::EmptyArray);
                    }
                    Some(JSONKey::Index(_)) if options.allow_trailing_commas => {}
                    _ => {
                        return_err!(ExpectedEndOfValue);
                    }
                }
                in_object = matches!(key_chain.last(), Some(JSONKey::Slice(_)));
            }
            Some(b'"') => {
                let rest = &on[idx..][1..];
                let Some(offset) = find_non_escaped_quoted(rest) else {
                    return_err!(ExpectedEndOfValue);
                };
                idx += offset + 2;
                emit!(RootJSONValue::String(JSONString(&rest[..offset])));
            }
            Some(b'0'..=b'9' | b'-') => {
                /// can include bad values
                fn find_non_number(chr: char) -> bool {
                    !matches!(chr, '0'..='9' | '.' | 'e' | 'E' | '+' | '-')
                }

                let rest = &on[idx..];
                let Some(offset) = rest.find(find_non_number) else {
                    return_err!(ExpectedEndOfValue);
                };
                idx += offset;
                emit!(RootJSONValue::Number(JSONNumber(&rest[..offset])));
            }
            Some(b't') if on[idx..].starts_with("true") => {
                idx += 4;
                emit!(RootJSONValue::Boolean(true));
            }
            Some(b'f') if on[idx..].starts_with("false") => {
                idx += 5;
                emit!(RootJSONValue::Boolean(false));
            }
            Some(b'n') if on[idx..].starts_with("null") => {
                idx += 4;
                emit!(RootJSONValue::Null);
            }
            Some(b @ (b',' | b'}')) if options.partial_syntax => {
                emit!(RootJSONValue::Empty);
                idx += 1;
                if in_object {
                    let _ = key_chain.pop();
                }
                if b == b',' {
                    continue;
                }
            }
            _ => {
                return_err!(ExpectedValue);
            }
        }

        while let Some(byte) = bytes.get(idx) {
            if key_chain.is_empty() {
                return Ok((idx, None));
            }

            if tls.is_some_and(|c: char| on[idx..].starts_with(c)) {
                idx += 1;
                if let [JSONKey::Index(ref mut idx)] = key_chain.as_mut_slice() {
                    *idx += 1;
                    break;
                }
            } else if let b' ' | b'\t' | b'\r' | b'\n' = byte {
                idx += 1;
            } else if let b'/' | b'#' = byte {
                let Some((comment, offset)) = parse_comment(&on[idx..]) else {
                    return_err!(InvalidComment);
                };
                idx += offset;
                if options.yield_comments {
                    emit!(RootJSONValue::Comment(comment));
                }
            } else {
                let new_byte = if *byte == b',' {
                    idx += 1;
                    if let Some(JSONKey::Index(ref mut idx)) = key_chain.last_mut() {
                        *idx += 1;
                    } else {
                        key_chain.pop();
                    }
                    if !options.allow_trailing_commas {
                        break;
                    }
                    skip_whitespace_find_comments!();
                    let Some(b @ (b'}' | b']')) = bytes.get(idx) else {
                        break;
                    };
                    *b
                } else {
                    *byte
                };
                match new_byte {
                    b'}' if *byte == b',' || in_object => {}
                    b']' if matches!(key_chain.last(), Some(JSONKey::Index(_))) => {}
                    _ => {
                        return_err!(ExpectedEndOfValue);
                    }
                }
                idx += 1;
                key_chain.pop();
                in_object = matches!(key_chain.last(), Some(JSONKey::Slice(_)));
            }
        }
    }

    let tl = tls.is_some_and(|_| matches!(key_chain.as_slice(), &[JSONKey::Index(_)]));

    if !(key_chain.is_empty() || tl) {
        return_err!(ExpectedBracket);
    }

    Ok((on.len(), None))
}

Some basics of the architecture

Callback function based (explained later)
No string copy (values have data based on points into the input buffer)
No dependencies

It supports many features

These are all described in the specification and are tested with spectra (a tool which allows checking pairwise code blocks representing an input and an output match the process of a program).

Usage

The foundations are defining a stack for the keys key_chain, when we find a value we run the call the cb with a reference to the key_chain and an enum variant instance with the value. For example, for finding values under the path ["message"]["message"] or ["message"]["level"] we can use pattern matching in the callback function.

use simple_json_parser::{
    JSONKey, RootJSONValue, parse_with_options as parse_json,
};

let _ = parse_json::<()>(&out, &Default::default(), |keys, value| {
	if let [
		JSONKey::Slice("message"),
		JSONKey::Slice(kind @ ("message" | "level")),
	] = keys
	{
		if let RootJSONValue::String(value) = value {
			println!("{kind}: {value}");
		}
	}
	None
});
use simple_json_parser::{
    JSONKey, RootJSONValue, parse_with_options as parse_json,
};

let _ = parse_json::<()>(&out, &Default::default(), |keys, value| {
	if let [
		JSONKey::Slice("message"),
		JSONKey::Slice(kind @ ("message" | "level")),
	] = keys
	{
		if let RootJSONValue::String(value) = value {
			println!("{kind}: {value}");
		}
	}
	None
});

This approaches avoids the need for a HashMap and allocating entries for values if we do not use them in the program.

Where JSON crops up

JSON comes up in written form as

package.json

And in data form

GitHub API
Cargo output
(spectra, depict)

Delimited JSON

The parser supports new-line-delimited JSON.

Unfortunately, this is not as simple as running the parser on each item of std::io::Lines because new-line-delimited JSON can be across multiple lines.

{ 
        "x": 2
}
{ 
        "y": 3, 
        "y2": 5 
}
{ 
        "z": 4
}
{ 
        "x": 2
}
{ 
        "y": 3, 
        "y2": 5 
}
{ 
        "z": 4
}

Building parsers

I am building a number of parsers at the moment, my goals with them are

Configurable: support as much syntax as possible. Use options at runtime to enable additional features
Lightweight
Low line of code count

Gotchas of the approach

The callback is needed (over a for-loop) because the Iterator trait is not designed to yield elements that based on the mutable lifetime given in Iterator::next. We could make it Iterator iterator but it would have to yield the owned Vec<JSONKey<'_>>, which would have allocate on each value (rather than use a single allocation). This is a limitation of the trait not for-loops. We could define a new trait IteratorWithLifetime which would have find but not collect etc. but still would not be usable in classic for-loops.

Gotchas of parsing JSON

This key-chain contextual parser has to also account for the fact that empty array values [] and empty object {} values have some meaning. Thus, we have emit!(RootJSONValue::EmptyObject);.

The parser currently uses a few macros, while normally I do not use them, because of control flow with the early returning it seemed acceptable to use them here. Maybe one day we can have reverse ? operations.

The parser has cut some small parts of correctness corners for the sake of remaining lightweight.

It does not check numbers. Again if you are trying to get a value from a JSON source as fast as possible, then there is little need to 1. Check the sequence of characters is valid 2. Parse it into a floating-point number.

fn find_non_number(chr: char) -> bool {
    !matches!(chr, '0'..='9' | '.' | 'e' | 'E' | '+' | '-')
}

let rest = &on[idx..];
let Some(offset) = rest.find(find_non_number) else {
    return_err!(ExpectedEndOfValue);
};
idx += offset;
emit!(RootJSONValue::Number(JSONNumber(&rest[..offset])));
fn find_non_number(chr: char) -> bool {
    !matches!(chr, '0'..='9' | '.' | 'e' | 'E' | '+' | '-')
}

let rest = &on[idx..];
let Some(offset) = rest.find(find_non_number) else {
    return_err!(ExpectedEndOfValue);
};
idx += offset;
emit!(RootJSONValue::Number(JSONNumber(&rest[..offset])));

Instead, we delegate parsing until usage time using the value_unwrap method on JSONNumber. We have the same for JSONString for the required unescape_string_content transform.

The parser also does not check for control characters in strings. I am not sure why this restriction is in the JSON specification? Maybe for a reason unrelated to JSON.

More on performance

A pivotal function used in JSON parsing is finding the end of a string. A simple method to achieve this would be

fn find_non_escaped_quoted(on: &str) -> Option<usize> {
	on.find('"')
}
fn find_non_escaped_quoted(on: &str) -> Option<usize> {
	on.find('"')
}

Unfortunately, the method for including double quotes in strings is not \q, but instead \". This means we have to check whether the last character was escaped. A naïve approach that accounts for this would be the following

fn find_non_escaped_quoted(on: &str) -> Option<usize> {
	let mut characters = on.bytes().enumerate();
	while let Some((idx, chr)) = characters.next() {
		if b'\\' == chr {
			let _ = characters.next();
		} else if b'"' == chr {
			return Some(idx);
		}
	}
	None
}
fn find_non_escaped_quoted(on: &str) -> Option<usize> {
	let mut characters = on.bytes().enumerate();
	while let Some((idx, chr)) = characters.next() {
		if b'\\' == chr {
			let _ = characters.next();
		} else if b'"' == chr {
			return Some(idx);
		}
	}
	None
}

But examining for the fact that a\\", a\\\\" and a\\\\\\" are all ends of a string but a\", a\\\" and a\\\\\" are not we can instead write the function as

fn find_non_escaped_quoted(on: &str) -> Option<usize> {
	on.match_indices('"').find_map(|(idx, _)| {
		let rev = on[..idx].bytes();
		let last = rev.rev().take_while(|chr: &u8| *chr == b'\\').count();
		(last % 2 == 0).then_some(idx)
	})
}
fn find_non_escaped_quoted(on: &str) -> Option<usize> {
	on.match_indices('"').find_map(|(idx, _)| {
		let rev = on[..idx].bytes();
		let last = rev.rev().take_while(|chr: &u8| *chr == b'\\').count();
		(last % 2 == 0).then_some(idx)
	})
}

This find_non_escaped_quoted function seems to be the bottleneck of the parsing implementation (when compared to other parts such as whitespace skipping etc.). Using depict we can compare the number of instructions run for each approach.

In first (with respect to the least instructions run) is the incorrect but future we would have had \q if un-escaped to " function.

symbol name	count
total	55834
`simple_json_parser::parse_with_options`	21173
`<core::str::pattern::CharSearcher as core::str::pattern::Searcher>::next_match`	9184
`simple_json_parser::parse_with_options::find_non_escaped_quoted`	3584
`parse::main`	142
`<std::env::Args as core::iter::traits::iterator::Iterator>::next`	78
`main`	7

The even check approach comes in second

symbol name	count
total	61142
`simple_json_parser::parse_with_options`	21173
`<core::str::pattern::CharSearcher as core::str::pattern::Searcher>::next_match`	9184
`simple_json_parser::parse_with_options::find_non_escaped_quoted`	8940
`parse::main`	141
`<std::env::Args as core::iter::traits::iterator::Iterator>::next`	78
`main`	7

With the naïve approach having executed six and half more AArch64 operations to find the end of the strings in my 5955 byte JSON example

symbol name	count
total	84201
`simple_json_parser::parse_with_options::find_non_escaped_quoted`	58528
`simple_json_parser::parse_with_options`	21188
`parse::main`	146
`<std::env::Args as core::iter::traits::iterator::Iterator>::next`	78
`main`	8

Comparing with a full JavaScript parser (Ezno)

As JSON is a subset of JavaScript objects, we can run the parser on the source. Here we see that the JSON uses 10x less instructions (55834 vs 664 254) that the full JavaScript parser (so ideally works out as 10x faster).

Obviously the JavaScript parser is doing more: actually parsing bytes in strings, allocating the parsed in vectors as an abstract syntax tree, storing source positions etc. But it is interesting to see the difference in a lightweight non-recursive parser vs a full JavaScript parser.

symbol name	count	mem_read	call
`Total`	664254	75258	6466
`ezno_parser::strings::parse_string`	108472	9552	225
`ezno_parser::lexer::Lexer::skip`	56382	20022	0
`<ezno_parser::expressions::object_literal::ObjectLiteralMember as ezno_parser::ASTNode>::from_reader`	32428	8174	1072
`ezno_parser::lexer::Lexer::skip_including_comments`	27404	6851	0
`ezno_parser::expressions::Expression::from_reader_after_first_expression`	16863	3773	140
`<ezno_parser::expressions::object_literal::ObjectLiteral as ezno_parser::ASTNode>::from_reader`	10727	4284	443
`ezno_parser::lexer::Lexer::is_immediate_keyword_advance`	9837	2526	309
`ezno_parser::expressions::Expression::from_reader_with_precedence`	9486	0	556
`ezno_parser::functions::MethodHeader::from_reader`	9380	1742	1072
`<ezno_parser::property_key::PropertyKey<U> as ezno_parser::ASTNode>::from_reader`	8910	2160	540
`<ezno_parser::comments::WithComment<T> as ezno_parser::ASTNode>::from_reader`	8335	30	2
`core::slice::memchr::memchr_aligned`	8234	754	0
`ezno_parser::lexer::Lexer::parse_string_literal`	6525	0	225
`ezno_parser::lexer::Lexer::expect`	6380	1450	145
`<alloc::borrow::Cow<str> as core::ops::arith::AddAssign<&str>>::add_assign`	5850	1575	0
`ezno_parser::lexer::Lexer::is_immediate_keyword`	4414	1104	158
`core::ptr::drop_in_place<ezno_parser::expressions::Expression>`	4296	1040	140
`ezno_parser::lexer::Lexer::is_keyword_advance`	4154	938	268
`ezno_parser::lexer::Lexer::is_operator_advance`	3908	809	269
`core::ptr::drop_in_place< ezno_parser::expressions::object_literal::ObjectLiteralMember >`	2948	670	0
`core::str::converts::from_utf8`	2791	393	0
`<T as alloc::slice::<impl [T]>::to_vec_in::ConvertVec>::to_vec`	2592	360	266
`core::ptr::drop_in_place< ezno_parser::comments::WithComment<ezno_parser::property_key::PropertyKey< ezno_parser::property_key::AlwaysPublic>>>`	2412	670	0
`__rustc::__rdl_alloc`	1785	0	0
`ezno_parser::numbers::parse_no_specifier`	1551	246	12
`<ezno_parser::expressions::Expression as get_field_by_type::GetFieldByType<source_map::span::BaseSpan<()>>>::get`	1072	268	0
`ezno_parser::lexer::Lexer::starts_with_function_header`	1040	52	130

This sort of side project work is not only useful for utilities, but simplifying the problem has recently helped to make Ezno faster through realising better ways to write it with prefix match style pattern and paying attention to where skip calls are made. (more details in the future).

Future

There is still some more things that could be improved in the future. SIMD may be useful in some places. The cb could yield a mutable reference to idx that could be used to skip over objects (with something of the form find_next_open_brace). Once that and a few more things are consolidated then the PR can be merged and these updates can be available to the crate on crates.io.

Any suggestions or feedback is welcome in the comments below!

Ideas on type inference

2025-09-26T00:00:00Z

In the third and final post of Ezno week, we will take a look at two kinds of type inference. One kind what I refer to as forward and the other as backward. One implemented and the other a work-in-progress.

The word inference in a type-checker commonly refers to creating (or finding) a type for a parameter (or variable in some cases). Often because the annotation is elided (jargon for omitted/missing). The inference process is a subset the general state of type synthesis, generating types from syntax.

//                         ↓ no type annotation here
const y = [1, 2, 3].map(item => item)
//                          	 ↑ refences to `item` resolves to type `number` here
//                         ↓ no type annotation here
const y = [1, 2, 3].map(item => item)
//                          	 ↑ refences to `item` resolves to type `number` here

Forward inference: pushing a type forwards

The first kind that I refer to forward is in partially annotated (or realised) source text. We have a function parameter without an annotation but we can figure a type out through looking at the context of where the containing function is written in.

If you have read the section on performance in the last post, you may have seen that TSC records a time for a bind pass but Ezno does not. I am not hiding that result, but infact the bind phase and check phase are merged in Ezno. Excluding the pass on top level declarations in a block used for hoisting, the AST is only walked once (compared to twice in TSC I think).

This can happen with and without annotations. If we declare a function as the value of variable that is annotated we need to push the annotation forward

//                                        ↓ no annotation here
const increment: (x: number) => number = x => x + 1;
//                      ↘----------------↗
//         we push the `number` parameter annotation forward
//                                        ↓ no annotation here
const increment: (x: number) => number = x => x + 1;
//                      ↘----------------↗
//         we push the `number` parameter annotation forward

But we also have cases where the user source is not annotated. For example in Promise.prototype.then

//                           ↓ no annotation here
fetch("...").then(function (res) {
//  using information about  ↓
//  what `fetch` returns     ↓
//  we can figure out that `res` is a `Response`
})
//                           ↓ no annotation here
fetch("...").then(function (res) {
//  using information about  ↓
//  what `fetch` returns     ↓
//  we can figure out that `res` is a `Response`
})

This is why I refer to it as forward. Using an annotation or other context we pass information forward in the source / buffer to figure out the type of the parameter.

The basics of the implementation

The implementation is quite simple, we pass an expected type through our synthesise_expression function.

fn synthesise_expression(
	expression: &ezno_parser::ast::Expression,
	// ↓
	expected_type: TypeId,
	// ↑
	environment: ..., 
	checking_data: ... etc
) -> TypeId
fn synthesise_expression(
	expression: &ezno_parser::ast::Expression,
	// ↓
	expected_type: TypeId,
	// ↑
	environment: ..., 
	checking_data: ... etc
) -> TypeId

So in the above example, we pass the resolved type in the variable

fn synthesise_variable_declaration(
	variable_declaration: &ezno_parser::ast::Expression,
	...
) {
	// ...
	let expected_type = synthesise_type_annotation(variable_declaration.type_annotation, /* ... */);
	let value = synthesise_expression(variable_declaration.expression, expected_type, /* ... */);
	// ...
}
fn synthesise_variable_declaration(
	variable_declaration: &ezno_parser::ast::Expression,
	...
) {
	// ...
	let expected_type = synthesise_type_annotation(variable_declaration.type_annotation, /* ... */);
	let value = synthesise_expression(variable_declaration.expression, expected_type, /* ... */);
	// ...
}

We do this passing in several places

Variable declarations pass their resolved annotation
Function arguments pass there respective parameter type
The resolved function return type annotation is stored in the environment and passed to the expression in return statements
satisfies passes its RHS annotation to the LHS. (so technically not forwards. Another to add to my gripes on the satisfies operator :weary:)
Class declarations with an implements clause (extends is contraversial)

When we arrive at a function or method declaration and this type is a function, we pass the expected parameter types forward to synthesise_function. When synthesising parameters if we arrive at a parameter without an annotation, we can pick it from the expected parameter.

Forward inference on properties gives unexpected member warnings for free

We do some transformation and passing in some cases. For example on object literals we get the property on the expected type

const myobj: { mymethod: (p: string) => string } = {
	// we evaluate `{ mymethod: (p: string) => string }["mymethod"]` to get the
	// expected type of `(p: string) => string`. The un-annoted parameter `p` 
	// therefore has a type of `string`
	//       ↓
	mymethod(p ) {
		p // ← string
	}
}
const myobj: { mymethod: (p: string) => string } = {
	// we evaluate `{ mymethod: (p: string) => string }["mymethod"]` to get the
	// expected type of `(p: string) => string`. The un-annoted parameter `p` 
	// therefore has a type of `string`
	//       ↓
	mymethod(p ) {
		p // ← string
	}
}

After adding the inference here, I had an an idea about how to add in the comment on an issue.

Thanks to the work of Patrick Laflamme who implemented this in PR #139 we can raise an early error for excess properties.

let property = environment.get_property(
	expected,
	Publicity::Public,
	&key,
	...
);

if property.is_none() {
	checking_data.diagnostics_container.add_warning(TypeCheckWarning::ExcessProperty { ... })
}
let property = environment.get_property(
	expected,
	Publicity::Public,
	&key,
	...
);

if property.is_none() {
	checking_data.diagnostics_container.add_warning(TypeCheckWarning::ExcessProperty { ... })
}

Currently implementation here

There are several decisions we take for the expected type for different kinds of expressions.

For assignments we use the constraint (variable or property type) on the LHS
For conditionals we pass the expected type to both branches
For object spreads we pass down the whole expected type

The last two assist in catching excess-property cases that for an unknown reason TypeScript doesn't catch.

Complications

One of the problems is around generics. Sometimes we might have a parameter that is based on generics.

For example addEventListener, the second function parameter is based of the first argument.

interface EventMap {
	"mousedown": MouseEvent
}

interface Node {
	addEventListener<T extends keyof EventMap>(name: T, cb: (ev: EventMap[T]) => void);
}
interface EventMap {
	"mousedown": MouseEvent
}

interface Node {
	addEventListener<T extends keyof EventMap>(name: T, cb: (ev: EventMap[T]) => void);
}

We can do this by specialising generics left-to-right before the next parameter. This is and some more and currently unimplemented.

One neat example I forgot I got working is computed generics! Here we get a union of the members, not just number. (more explanation in the future).

const x = [1, 2, 3];
x.map(a => (a satisfies string, 2)) // Expected string, found 1 | 2 | 3
const x = [1, 2, 3];
x.map(a => (a satisfies string, 2)) // Expected string, found 1 | 2 | 3

However, there are cases where forward is not enough

Wrong direction

One of the problems with this simple forward pass is when things are not forwards.

For example if we have functions of the form

function func<T>(cb: (ev: EventKind[T]) => void, kind: T) {}

func((ev) => {}, "mousedown")
function func<T>(cb: (ev: EventKind[T]) => void, kind: T) {}

func((ev) => {}, "mousedown")

We do not know the expected type of the first paramter, until we have looked at the second parameter.

Similarily if we write our function outside of the context of usage. Because forwards runs in the same direction as checking we cannot infer ev here until afterwards.

function doWhenMouseDown(ev) {}

document.addEventListener("mousedown", doWhenMouseDown);
function doWhenMouseDown(ev) {}

document.addEventListener("mousedown", doWhenMouseDown);

While the forward single pass has a ceiling, there are ways to performance inference above this.

Backward inference

This is the harder and still work-in-progress kind of inference. It is also something new, an interpolation inside of existing TypeScript.

Where forwards involves context on the written parameter, backwards involves usage of the reference to the parameter.

function sin(parameter: number) { 
	// ...
}

function func(param) {
	const property = parameter.value; // require the `parameter` to have a property under key `value`
	return sin(property) // require `parameter.property` to have type `number`
}
function sin(parameter: number) { 
	// ...
}

function func(param) {
	const property = parameter.value; // require the `parameter` to have a property under key `value`
	return sin(property) // require `parameter.property` to have type `number`
}

This feature has been implemented before in the Hegel type checker.

The idea here is that usage of the value adds requirements to the constraint of the parameter (or any of its descendents).

You can see attempt #2 working in a branch I started last year.

This is attempt #2 at the feature. Attempt #1 was partially working 3 years ago, but the implementation required mutating the parameter type during checking of the body, which caused all sort of problems. This new one pushes a sequence of requirement or constraints to the current context. At the end of checking the body, these requirements are collapsed into a final type.

Steps for backward inference landing in `0.1.0`

While three of the fundamental tests are passing, there is still a bit more to do to move this from a demonstration to something that could make writing real-world type-safe code easier.

The first is to collect more examples of untyped code, to see whether there are more cases to collect and implement.

One of those is the cases is free-variables. Ezno does not infer constraints for variable in standard flow (which is a whole other topic). But when a variable crosses a function boundary, there is some special handling under the title of free-variables.

let a = null;

function initialiseA() {
	a = data()
}

function doThing() {
	const x: string = a
}
let a = null;

function initialiseA() {
	a = data()
}

function doThing() {
	const x: string = a
}

Here doThing requires a to be string (not null). Some assignment needs to happen before any calls to doThing.

The inference can treat find requirements for free variables (treating them the same as parameters), but currently attempt #2 checking for these constraints. Attempt #1 did do this but I have concerns about performing this eagerly...

Another is conditionals, this is something attempt #1 could not achieve because of the mutations. Here item should infer as { tag: "a", data: number } | { tag: "b", data: string } rather than { tag: any, data: string & number }.

if (item.tag === "a") {
	const x: number = a.data;
} else (item.tag === "b") {
	const y: string = a.data;
}
if (item.tag === "a") {
	const x: number = a.data;
} else (item.tag === "b") {
	const y: string = a.data;
}

Allowing valid JavaScript, with new intrinsics

One of the aims of this feature is to allow JavaScript code to be checked. Here we want to catch false-negatives, errors that occur at runtime but are uncaught by the checker. But we also have to consider this could introduce false-positives. For example Math.sin("2.2") is totally valid, the string "2.2" can be passed and we do not get an error (Infact it returns 0.808... as the function performs an implict cast). In fact any value passed to the Math.sin is valid (unless its toPrimitive throws). In the following, one argument is a number type and another is not, yet we get identical output...

Math.sin(1/0) // NaN
Math.sin(new (class X {})) // (also) NaN
Math.sin(1/0) // NaN
Math.sin(new (class X {})) // (also) NaN

Currently the second call is outlawed by the rules TypeScript, which technically does not make a JavaScript superset...

In a previous post I outlined some intrinsic types I added to the checker. These are types different to the standard root, union, interesection, generic etc types in that they have specific behavior for a situation. Most of them narrow down on some type, such as MultipleOf that only allows numbers that are of some multiple to be passed. Some like TSCs NoInfer controls behavior around inference.

So a solution we could invent here is a new intrinsic type called Warn. This could have a type and tag associated with it. When the checker see that the RHS of the subtype matches the first type argument we could allow it but also emit an error that it was matched.

type NumberLike = number | Warn<any, "implicit cast">;
//                         ^^^^

interface Math {
	sin(x: NumberLike): number;
}

declare const Math: Math;

// Warning (not error): "2.2" is allowed but is a "implicit cast"
Math.sin("2.2") 
type NumberLike = number | Warn<any, "implicit cast">;
//                         ^^^^

interface Math {
	sin(x: NumberLike): number;
}

declare const Math: Math;

// Warning (not error): "2.2" is allowed but is a "implicit cast"
Math.sin("2.2")

Any suggestions: for the name, usage etc are welcome on this issue.

Then anything inferred from the argument passing would be valid as it currently can create a false posiitive.

Wrapping up

There are still somethings to say about what backward inference can solve, but I will cut it here on the ideas and share them once it works.

You can read about when the next release will be in the previous blog post.

If you want to see more of this kind of thing you can follow me on x, bluesky and join the existing great sponsors of the project!.

Specification, speed and (a) schedule

2025-09-24T00:00:00Z

It is Ezno week! As it has been three years since the initial announcement I thought I would publish three blog posts that were lying around and trying to spin this project up again after taking a break this last year.

To recap Ezno is "A fast and correct TypeScript type checker and compiler with additional experiments".

I started this project because: I wanted to learn more Rust (and to get better at programming on large projects) and I had some interesting ideas about what a type-checker could do.

Also because there were still Coronavirus lockdowns around that time and that was the only thing to do back then.

Three years since the announcement: it has gone from a private demo on a collection of ideas, to a small open-source project. Development currently has been slowly chipping away at sections of a feature, getting some base cases working, understanding the core abstractions and then experimenting with new ideas.

Although it is not ready for production, there are many achievements to celebrate at this checkpoint stage. This post outlines an overview of some progress since the announcement.

Specification

At the announcement, there were no tests for type-checking. Testing a large and continually changing project can unrewarding and laborious. In 2023, I added tests to the project in the form of a markdown file that is compiled in to Rust code using the default test harness. This markdown form of tests for type-checking doubles up as readable documentation for the currently supported features as well as being a format that is really easy to add new tests to.

In the year after, working on more features the number of tests went from around 30 to around 350.

The latest iteration of this markdown-testing idea has been in a standalone tool called spectra. It is currently still as an experiement or a canary release but I have been using it with great success on several of my parsers.

With some recent improvements, there are now 356 tests passing on Ezno.

The next step is a v2 of the specification with general improvements to split the specification.md into multiple files and add more detailed documentation into it.

If you have suggestions for documentation, things you think would be good to include you can leave comments on the PR.

Speed

Building a performant type-checker is a hard task. To get the type systems working in the first place is a challenge on its own but to do that while considering memory usage and eliminating overhead is a whole other level.

However, it is a benefitial thing to do. For users it can reduce CI costs and buys more space for additional features.

And as an implementor it is quite an interesting, you learn about lower-level tooling and forces you to revisit previous code.

Oftenly making code shorter and more simple rewards you with the improved performance without thinking about the details.

This has been the case with last years parser change. Moving to a simpler model has brought the parsing way down. Before the change, for large files, the parsing time count was 2-3 times the type-checking time. After the change the parsing took the same time as type-checking.

More recently (at the start of this summer), I worked on optimising the layout of structures in the parser. I built a parser for some of the diagnostics you can get out of the Rust compiler. Using this tool I could easily find some places where there were enums with variants significantly larger than common constructed forms (clippy does not catch all cases). After some changes I managed to drop memory usage by over half as well as a ~20% reduction in execution time. Comparing these currently unmerged changes against the main branch, the CI reports.

binaries/binary-LinuxX64-main-last/binary-LinuxX64-main-last: 10 837 025 bytes allocated and 44 930 722 instructions
binaries/latest/ezno: 4 660 067 bytes allocated and 39 699 629 instructions
binaries/binary-LinuxX64-main-last/binary-LinuxX64-main-last: 10 837 025 bytes allocated and 44 930 722 instructions
binaries/latest/ezno: 4 660 067 bytes allocated and 39 699 629 instructions

and

Summary
  ./binaries/binary-LinuxX64-general-fixes-last/binary-LinuxX64-general-fixes-last check demo.tsx ran
  ...
  1.21 ± 0.02 times faster than ./binaries/binary-LinuxX64-main-last/binary-LinuxX64-main-last check demo.tsx
Summary
  ./binaries/binary-LinuxX64-general-fixes-last/binary-LinuxX64-general-fixes-last check demo.tsx ran
  ...
  1.21 ± 0.02 times faster than ./binaries/binary-LinuxX64-main-last/binary-LinuxX64-main-last check demo.tsx

This is one important part the performance work: to have a open and public log of results that can calculate differences and integrates with the CI checking and so forth.

GitHub actions is far from perfect, I have been trying to find a more precise and predictable tool to hyperfine (ironically). GitHub actions can vary in performance quite a bit and do not have access to systems that require root level access such as perf_events.

Last Friday, I found out about SDE. I am still learning about it, but effectively it does binary instrumentation (effectively a CPU emulator) to run a binary while counting various kinds of instructions that have been run. This is perfect as we can see exact counts of instructions run (not emitted) and under different sections of the program. Of course I then built a parser to analyse the output and now have it running in the GitHub CI.

I am still learning what it emits and how to act on it. But recent work on another project found Rust does not collapse a hot path in the lexer. On my WIP branch I managed to change some things in the lexer to take total instructions 28 354 432 instructions to 27 286 219.

I will do a dedicated blog post on the parser and the exacts of last years lexer change but there is still more work to do. Similiarly for "monitoring performance in CI", I still need to learn more about SDE. Whether it can give more information on memory, why it lists 10 million less instructions run than callgrind, dealing with inlined code, instruction wall-clock time heuristics and maybe adding a 'diffing' functionality to my sde-parsing tool. If you have tips/experience, leave it in the comments below!

This is on top of a the existing general principle of the project of writing minimal code, using a low numbers of dependencies, writing iterators rather than allocating vectors etc.

So with all these improvements, how is the performance going?

Well using the specification.md file from earlier we can build an amalgamation of examples on the specification. We remove import and export examples as we want to compile it to a single file and we do some transformations so there are no clashes with names. The corpus contains a breadth of features (and so tests a wide range of behaviors that could exist in a program) and now with 356 (minus module tests) we get around 2k LOC of code to test against. It doesn't accurately reflect a program, but it alright at this stage. We know running performance results on this file are accurate because the examples covers all features and does not include stuff that is incomplete or can currently crash the checker. You can view the whole file here.

And running in the new CI setup, we see the latest on the general-fixes branches runs in around ~10ms.

This does not mean much on its own, so we can compare it to the most complete type-checkers and here we see an impressive result when checking our 2k LOC example.

Benchmark 1: ./ezno check demo.tsx
  Time (mean ± σ):       8.3 ms ±   0.2 ms    [User: 6.3 ms, System: 1.7 ms]
  Range (min … max):     8.0 ms …   9.5 ms    316 runs
 
Benchmark 2: ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo.tsx
  Time (mean ± σ):      92.1 ms ±   3.5 ms    [User: 119.0 ms, System: 29.3 ms]
  Range (min … max):    86.3 ms … 100.4 ms    32 runs
 
Benchmark 3: node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo.tsx
  Time (mean ± σ):     772.9 ms ±   6.3 ms    [User: 1451.6 ms, System: 63.3 ms]
  Range (min … max):   761.9 ms … 780.4 ms    10 runs
 
Summary
  ./ezno check demo.tsx ran
   11.16 ± 0.48 times faster than ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo.tsx
   93.65 ± 2.18 times faster than node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo.tsx
Benchmark 1: ./ezno check demo.tsx
  Time (mean ± σ):       8.3 ms ±   0.2 ms    [User: 6.3 ms, System: 1.7 ms]
  Range (min … max):     8.0 ms …   9.5 ms    316 runs
 
Benchmark 2: ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo.tsx
  Time (mean ± σ):      92.1 ms ±   3.5 ms    [User: 119.0 ms, System: 29.3 ms]
  Range (min … max):    86.3 ms … 100.4 ms    32 runs
 
Benchmark 3: node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo.tsx
  Time (mean ± σ):     772.9 ms ±   6.3 ms    [User: 1451.6 ms, System: 63.3 ms]
  Range (min … max):   761.9 ms … 780.4 ms    10 runs
 
Summary
  ./ezno check demo.tsx ran
   11.16 ± 0.48 times faster than ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo.tsx
   93.65 ± 2.18 times faster than node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo.tsx

When we concatenate the file 10 times over, we see a good but slightly less impressive result compared to TSC based compilers (for some reason they does not seem to scale linearly).

Benchmark 1: ./ezno check demo10.tsx
  Time (mean ± σ):      72.3 ms ±   1.3 ms    [User: 60.3 ms, System: 11.7 ms]
  Range (min … max):    70.3 ms …  78.7 ms    41 runs
 
Benchmark 2: ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo10.tsx
  Time (mean ± σ):     476.2 ms ±   9.8 ms    [User: 646.7 ms, System: 69.9 ms]
  Range (min … max):   464.1 ms … 496.4 ms    10 runs
 
Benchmark 3: node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo10.tsx
  Time (mean ± σ):      1.863 s ±  0.086 s    [User: 4.296 s, System: 0.104 s]
  Range (min … max):    1.805 s …  2.082 s    10 runs
 
Summary
  ./ezno check demo10.tsx ran
    6.58 ± 0.18 times faster than ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo10.tsx
   25.76 ± 1.27 times faster than node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo10.tsx
Benchmark 1: ./ezno check demo10.tsx
  Time (mean ± σ):      72.3 ms ±   1.3 ms    [User: 60.3 ms, System: 11.7 ms]
  Range (min … max):    70.3 ms …  78.7 ms    41 runs
 
Benchmark 2: ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo10.tsx
  Time (mean ± σ):     476.2 ms ±   9.8 ms    [User: 646.7 ms, System: 69.9 ms]
  Range (min … max):   464.1 ms … 496.4 ms    10 runs
 
Benchmark 3: node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo10.tsx
  Time (mean ± σ):      1.863 s ±  0.086 s    [User: 4.296 s, System: 0.104 s]
  Range (min … max):    1.805 s …  2.082 s    10 runs
 
Summary
  ./ezno check demo10.tsx ran
    6.58 ± 0.18 times faster than ./node_modules/@typescript/native-preview-linux-x64/lib/tsgo --noEmit --jsx preserve --skipLibCheck demo10.tsx
   25.76 ± 1.27 times faster than node ./node_modules/typescript/lib/_tsc.js --noEmit --jsx preserve --skipLibCheck demo10.tsx

Full benchmark public here

With --timings in Ezno and --diagnostics in TSC. We can see some a breakdown for these results

# in Ezno
Diagnostics: 4320
Types:       48667
Lines:       21990
Cache read:  205.976µs
FS read:     1.383218ms
Parsed in:   23.138385ms
Checked in:  37.645912ms
Narrowing:   327.594µs
Reporting:   639.726µs

# in tsgo
Files:               8
Lines:           61095
Identifiers:     70224
Symbols:         50955
Types:           11370
Instantiations:   4224
Memory used:    42754K
Memory allocs:  793413
Parse time:     0.059s
Bind time:      0.021s
Check time:     0.392s
Emit time:      0.000s
Total time:     0.509s
# in Ezno
Diagnostics: 4320
Types:       48667
Lines:       21990
Cache read:  205.976µs
FS read:     1.383218ms
Parsed in:   23.138385ms
Checked in:  37.645912ms
Narrowing:   327.594µs
Reporting:   639.726µs

# in tsgo
Files:               8
Lines:           61095
Identifiers:     70224
Symbols:         50955
Types:           11370
Instantiations:   4224
Memory used:    42754K
Memory allocs:  793413
Parse time:     0.059s
Bind time:      0.021s
Check time:     0.392s
Emit time:      0.000s
Total time:     0.509s

Again Ezno is currently work-in-progress, so these differences are not usable in code today. But you can see that currently implemented parts are going well.

tsgo is great, because it builds something I did not exactly want to build that being: a exact drop-in faster type-checker for use in the near future. It never made any sense to try and do what is the job of the TSC team for exclusively performance reasons, which is something I have not been too bothered by when writing TypeScript code.

Instead, I decided to build something different, new and original. Based on learning how it all works, rather than being a human transpiler. Recognising that some of TSCs code has been written a while ago, with different aims. That there are new features that can be added in places and existing problems can be fixed by starting from the ground up. And that at the end of the day you can still maintain some compatability and keep existing features.

There are a few differences between the type-checkers, firstly the checking behaviour is different in Ezno compared to TSC (with an aim to get very similar errors). Ezno is written in Rust whereas tsgo is written in Golang. Ezno does not doing anything in parallel currently, but may when it is more mature. For lib.d.ts it has a compressed binary form, rather than reading a pseudo-human readable file. This drops a lot of initial overhead as it does not have to deal with hoisting etc. Without --max-diagnostics being specified, Ezno only prints the first 30 diagnostics whereas tsc prints every error.

Rust and Golang, are very good programming languages especially when building compilers, parsers and other command line programs. It is clear why the TSC team went for Golang with its inbuilt mark-and-sweep garbage collection for there syntactical port. On the other hand, Ezno will continue with its Rust implementation.

A schedule?

Not quite the right heading, but have to stick to the 's' theme

This blog post has been a showing a trend in the right direction on the specification tests and performance numbers. This is a hobby project for me, not a job. It has been an extracurricular, in learning new things and building stuff that I can direct and deem new and useful. I do not set many deadlines and work mostly on things that I think will be fun to work on.

Events/partial application is the invisible elephant in the room for usability. I would like to do a talk before the blog post, so if you have meetup in Europe get in contact! I would like to release the blog post on some of the findings within the twelve months. While cool, the events feature is large and complex, which will be hard to scale to large and complex codebases. Before the next 0.1.0 release there will be a flag to disable some of the feature and make using external packages infinitely easier.

For the framework idea proposed in the initial blog post, it has not really been worked on since 2022. The improvement to the underlying compiler findings has made it a lot more possible and flexible, so maybe when I find the time I will put out a demonstration of how the type-checking logic can benefit specific parts of front-end code. But it has moved out of the primary focus of the compiler.

There are three other ideas I want to work on for version 0.1.0. All going well it could be as soon as the end of this year? The 0.1.0 release should work (aka catch 95% of type erros) in small low-dependency projects. So I will cut the release when I have some of my examples working at that level (and not just concatenations of test cases).

You can star and watch the repository for that announcement and sponsor to help it fall this side of the new year!

For more updates you can follow this as blog as RSS and follow me on X and BlueSky.

Implementing narrowing

2025-09-22T00:00:00Z

This post is on implementing the type-narrowing feature from TypeScript into my own type-checker ezno.

What is type narrowing?

TypeScript is an extension to JavaScript. In JavaScript every value has a type.

console.log(typeof 4) // number
console.log(typeof "hiya") // string
console.log(typeof false) // boolean
console.log(typeof { a: 2 }) // object
console.log(typeof (() => 3)) // function
console.log(typeof 4) // number
console.log(typeof "hiya") // string
console.log(typeof false) // boolean
console.log(typeof { a: 2 }) // object
console.log(typeof (() => 3)) // function

We can reason about values and variables before run-time using type checkers and types annotations.

print_type(2) // 2

function func(param: string) {
	print_type(param) // string
}
print_type(2) // 2

function func(param: string) {
	print_type(param) // string
}

There are many places where types annotations can exist. These annotations can inform the checker about the shapes of the values a reference can have so that the checker program can report errors and more ahead of time.

There are other ways types are realised within the checker aside from annotations. One of them is the information present in the assumption of a condition holding in a branching structure.

Here we see that because of the condition param === "hi", we can narrow in on the type string to knowing that value of param in the branch is a constant string "hi".

function func(param: string) {
	if (param === "hi") {
		print_type(param) // "hi"
	}
}
function func(param: string) {
	if (param === "hi") {
		print_type(param) // "hi"
	}
}

Why type narrowing?

This functionality enables a few things

1. We can branch on the tags of sum types

type Kinds =
	| { tag: "person", name: string, age: number }
	| { tag: "vehicle", horsepower: number }

function print_kind(item: Kinds) {
	if (item.tag === "vehicle") {
		print_type(item.horsepower)
	}
}
type Kinds =
	| { tag: "person", name: string, age: number }
	| { tag: "vehicle", horsepower: number }

function print_kind(item: Kinds) {
	if (item.tag === "vehicle") {
		print_type(item.horsepower)
	}
}

2. We can filter out null values through early returns

function print(item: MyObject | null) {
	if (item !== null) {
		return
	}
	console.log(item.name)
}
function print(item: MyObject | null) {
	if (item !== null) {
		return
	}
	console.log(item.name)
}

and through short-circuiting operators

typeof x === "string" && x.contains("blue")
typeof x === "string" && x.contains("blue")

3. We can detect exact values (here through a free-variable)

let x: string = "";

function print() {
	if (x === "hello") {
		// ...

		// Warning here!
		if (x === "hello") {
		
		}
	}
	
}
let x: string = "";

function print() {
	if (x === "hello") {
		// ...

		// Warning here!
		if (x === "hello") {
		
		}
	}
	
}

4. Checking data exterior to the program

const item = await fetch("./my_api.json");

if (typeof item.property === "string") {
	console.log(item.property);
}
const item = await fetch("./my_api.json");

if (typeof item.property === "string") {
	console.log(item.property);
}

Effectively it enables

Using properties of data of varying structure
Using properties of data of unknown types
Catching truthy values

The idea behind narrowing is to pick a new type (that is a subset of the current type) based on properties enforced by the condition.

We can do that given we have in the context

Some unknown type: parameter or from some external source.
A conditional expression

Information on types

Now we know what narrowing is and why it exists as a feature, we can see how we may go about implementing it.

Ezno is not a rewrite or port of the TypeScript Compiler (TSC), so this implemention was built on my own intuition and may be completely different to how it is built in TSC. We can see thought that it matches much of the same functionality of TSC.

The first thing to know is how Ezno represents types. Here parameter types and external types are wrapped in special types similar to generics. These wrapper types therefore have a flag and as they are new types, a unique TypeId.

#5: string
...
#1232: Parameter { name: "param", backing: #5 }
#5: string
...
#1232: Parameter { name: "param", backing: #5 }

When we use a parameter type we add more information onto it. These information is achieved under Constructor. So param === "hi" and param.length become Constructor::Operation { operation: StrictEqual, lhs: #1232, rhs: *"hi"* } and Constructor::Property { lhs: #1232, rhs: *"length"* } respectively. We can reduce these constructors to their base types of boolean and number when using them but storing them, we hold onto these richer properties.

Narrowing: the implementation

The main narrow function takes the type of the expression and returns a list of new constraints for various values.

For example given our above example, we have something like these (described in TypeScript rather than Rust).

function narrow(expression: Parameter | Constructor) -> Map<Id, Type>;

narrow(*c == 3*) -> [[*c*, Constant(3)]]
function narrow(expression: Parameter | Constructor) -> Map<Id, Type>;

narrow(*c == 3*) -> [[*c*, Constant(3)]]

Producing a narrowed map is implemented in the single file checker/src/features/narrowing.rs.

The narrow implementation is quite simple: we take our Parameter and Constructor objects and figure out narrowed type-to-type key-value entries in our returned map.

Operator `Constructor`s

There are many kinds of expressions that give more information about values

a // a is truthy
a === ? // a now has the type of the RHS
typeof a == ? // a now has a type defined by 
a[prop] == ? // a has prop of type
prop in a // a has a prop
a instanceof ? // a is an object with prototype
a // a is truthy
a === ? // a now has the type of the RHS
typeof a == ? // a now has a type defined by 
a[prop] == ? // a has prop of type
prop in a // a has a prop
a instanceof ? // a is an object with prototype

Here each has types that are structured as

*parameter*
Constructor::Equal { lhs: *parameter*, rhs: ? }
Constructor::Equal { lhs: Constructor::TypeOf { operand: *parameter* }, rhs: ? }
Constructor::Equal { lhs: Constructor::Propery { operand: *parameter*, property: *prop* }, rhs: ? }
Constructor::HasProperty { lhs: *parameter*, rhs: *prop* }
Constructor::InstanceOf { lhs: *parameter*, rhs: ? }

When we are passed one of these constructors we can return a key value pair. There is more going on but it is almost as simple as the following

fn narrow(ty: TypeId) -> HashMap<TypeId, TypeId> {
	// ...
	if let Constructor::InstanceOf { lhs, rhs } = ty_object {
		return HashMap::from_iter([(lhs, rhs)]);
	}
	// ...
}
fn narrow(ty: TypeId) -> HashMap<TypeId, TypeId> {
	// ...
	if let Constructor::InstanceOf { lhs, rhs } = ty_object {
		return HashMap::from_iter([(lhs, rhs)]);
	}
	// ...
}

While roots are relatively simple, we have to do a bit more when the lhs is a union type.

Union filtering

If we have x: string | number (a union type) and a check of typeof x === "string", we want to find types from the [string, number] vector that pass the check (in this case string).

Union types are represented as trees, so we do a recursive walk yielding types that pass the filter to a found array.

There are (currently) six kinds of filter

HasProperty
HasPrototype
IsType
null | undefined
Falsy
Not<Filter>

These each yield true or false for each type passed by the filter. If true we add that field to vector to build the new, filtered union type.

Logical operations: appending ands

Conditions can be combined. One of those is the logical and &&, which can requires the two operands to hold.

To narrow these we collect each side and concatenate the left cases with the right cases. Here isNumber(a) -> a is number and isString(b) -> b is string get combined to yield a narrowed result of isNumber(a) && isString(b) -> [a is number, b is string]

if (isNumber(a) && isString(b)) {
	a // is number
	b // is string
}
if (isNumber(a) && isString(b)) {
	a // is number
	b // is string
}

Logical operations: or

The other logical operator is the inclusive or: ||. This requires a bit more handling. The following if body can run with a is boolean and b is string therefore we can make no conclusions about a (and vice versa).

if (isNumber(a) || isString(b)) {
	a // is ?
	b // is ?
}
if (isNumber(a) || isString(b)) {
	a // is ?
	b // is ?
}

Here we find the keys of the narrowed maps to be disjoint and so produce an empty narrowed map

However, if both operands of the logical operator yield a key-value pair for the same LHS parameter type we can union the results.

if (isNumber(x) || isString(x)) {
	x // is number or x is string
}
if (isNumber(x) || isString(x)) {
	x // is number or x is string
}

x is number --\
			   |---> x is number | string
x is string --/			   
x is number --\
			   |---> x is number | string
x is string --/

You can see this merging here

Logical operators: negation operators

The final logical operation is negation. We control this through an additional parameter in our narrow function. When the negate parameter is true it mostly disables narrowing, but as we will see later we it can be useful with additional intrinsic types.

function func(x: number) {
	if (x !== 3) {
		x // is not 3?
	}
}
function func(x: number) {
	if (x !== 3) {
		x // is not 3?
	}
}

Negation is not always through the bash (!) operator. We sometimes see it crop up implicitly through else branches.

if (isNumber(x)) {
	// ...
} else {
	x // is not number
}

// as this is equivalent to

if (isNumber(x)) {
	// ...
}
if (!isNumber(x)) {
	x // is not number
}
if (isNumber(x)) {
	// ...
} else {
	x // is not number
}

// as this is equivalent to

if (isNumber(x)) {
	// ...
}
if (!isNumber(x)) {
	x // is not number
}

Logical operators: De Morgan's laws

We can combine negation with the && and || operators. Those familiar with De Morgan's laws may know what we can do here.

If we have a !(a || b), we can treat it as !a && !b. So for the following we find the following value for x in the else branch

if (x === 2 || x === 10) {

} else {
	x // is not 2 and not 10
}
if (x === 2 || x === 10) {

} else {
	x // is not 2 and not 10
}

This can be seen here

We have the same logic for treating !(a && b) as !a && !b

Aside: logical operation representation

For those who may be digging through the code wondering where Constructor::LogicalOr or Constructor::LogicalNegation is. There is a interesting short-cut taken by the compiler in representing these types.

We do this by realising the following.

!b     ≡ b ? false : true;
x || y ≡ x ? x : y;
x && y ≡ x ? y : x;
!b     ≡ b ? false : true;
x || y ≡ x ? x : y;
x && y ≡ x ? y : x;

So the actual represention of these types is Constructor::ConditionalResult.

Sometimes we want this lifted form, so there are helper methods for finding these.

Control flow

As mentioned above, when checking the else branch, we take the condition in the truthy and run narrowing again with negate = true this time.

// ↓ ↓ ↓
let negate = true;
// ↑ ↑ ↑
let values = super::narrowing::narrow_based_on_expression_into_vec(
	condition,
	// passed here
	negate,
	environment,
	&mut checking_data.types,
	&options,
);
// ↓ ↓ ↓
let negate = true;
// ↑ ↑ ↑
let values = super::narrowing::narrow_based_on_expression_into_vec(
	condition,
	// passed here
	negate,
	environment,
	&mut checking_data.types,
	&options,
);

This may be duplicate type scanning and maybe could be made faster in the future

Building new objects

For the following

function func(obj: any) {
	if (typeof obj.property === "number") {
		// ...
	}
}
function func(obj: any) {
	if (typeof obj.property === "number") {
		// ...
	}
}

We build up a new object { property: number } for obj.

What to do with the narrowed values?

Now we have a map of narrowed values, we then store these on the context local to the branch body.

At several stages, when using the reference, we swap out the backing type with a narrowed value from the context.

pub(crate) fn get_value_of_variable(...) {
	/// ...
	let current_value = current_value.or_else(|| fact.variable_current_value.get(&on).copied());
	
	if let Some(current_value) = current_value {
		// info = property on context
		let narrowed = info.get_narrowed_or_object(current_value, types); 
		if let Some(narrowed) = narrowed {
			return Some(narrowed);
		} else {
			return Some(current_value);
		}
	}
	/// ...
}
pub(crate) fn get_value_of_variable(...) {
	/// ...
	let current_value = current_value.or_else(|| fact.variable_current_value.get(&on).copied());
	
	if let Some(current_value) = current_value {
		// info = property on context
		let narrowed = info.get_narrowed_or_object(current_value, types); 
		if let Some(narrowed) = narrowed {
			return Some(narrowed);
		} else {
			return Some(current_value);
		}
	}
	/// ...
}

Thus concluding how narrowing is implemented:

Have types that store information about operators etc
Visit the condition type, producing a map of narrowed constraints. Apply logical operator rules and run filters
Store that information in the branch context
When looking up a type of variable etc, look for a narrowed type

More narrowing

We have covered that basics of narrowing. Here are some other interesting things I also implemented.

Explicit type annotation type guards

While we can write expression type guards, sometimes this functionality is hidden behind functions. To do this we have return type guards.

function isNumber(x: any): x is number {
  return typeof x === "number";
}
function isNumber(x: any): x is number {
  return typeof x === "number";
}

This works simply again with x being an implict generic. For x is y type annotations, we look up the parameter type of x and use that type. At usage, this is done by specialisation like other generics.

The checker has not be tested on schema libraries like arktype or zod which use return type guards. Other features of the type system are missing to support this

These type guards are synthesised as annotations into a constructor not producible from JavaScript expressions (at the moment 👀). However, because of the Constructor type, we get inferred type guards for free.

We also can check them (a feature TypeScript does not currently have).

This is implemented in type subtyping

Inference of `Not` intrinsic

Ezno has additional types that can represent things not possible in TypeScript. One of those is the Not<Type>. You can read more about these additional types in a previous post.

With the Not intrinsic generated by narrowing we can find invariants in the program.

Inference of other number intrinsics

We can also do this for numbers.

Again find out more about these additional types in the previous 'experimental types' post.

Performance and type overhead concerns

Current benchmarking shows roughly 1% of checking time is on building the narrowing map.

Diagnostics: 4320
Types:       48667
Lines:       21990
Cache read:  231.702µs
FS read:     1.620879ms
Parsed in:   28.493175ms
Checked in:  43.683541ms
Narrowing:   400.151µs
Reporting:   903.007µs
Diagnostics: 4320
Types:       48667
Lines:       21990
Cache read:  231.702µs
FS read:     1.620879ms
Parsed in:   28.493175ms
Checked in:  43.683541ms
Narrowing:   400.151µs
Reporting:   903.007µs

The extra intermediate types with deep properties may be a general concern though for performance.

Wrapping up

Hopefully this was interesting short explanation on how I implemented type narrowing. More posts coming soon!

II lines of code

2025-08-14T00:00:00Z

I have been recently working on a few one-off random libraries and tools. Here are a few interesting findings from building one of those: a library for formatting and parsing various representations of numbers.

This post covers some representations for numbers and how we can write a formatting function u32 -> String and (the partial inverse of that function) that parses the format String -> u32.

The term "number" in the post refers to a 32-bit or 64-bit fixed point unsigned integer. We will not be covering negative or fractional values for now. Additionally, the theory and logic is the focus of the post but the code and library is written in Rust.

Base two and base ten

Computers work on electricity. They store and transmit numbers through high and low voltages.

There are two states high and low. This is known as binary with bi the numeric prefix for two. We store each part as a bit. We can store zero as low 0 and one as high 1. However, we see the successive number two cannot be represented in one signal. So it becomes in a sequence where we start with so we shift over to the left. 10. The general pattern is when we add one to a something where the right starts with a zero, we set all trailing ones to zero and set one before said block.

> ./format-number 0 --binary         > ./format-number 6 --binary
0                                    110
> ./format-number 1 --binary         > ./format-number 7 --binary
1                                    111
> ./format-number 2 --binary         > ./format-number 8 --binary
10                                   1000
> ./format-number 3 --binary         > ./format-number 9 --binary
11                                   1001
> ./format-number 4 --binary         > ./format-number 10 --binary
100                                  1010
> ./format-number 5 --binary         > ./format-number 11 --binary
101                                  1011
> ./format-number 0 --binary         > ./format-number 6 --binary
0                                    110
> ./format-number 1 --binary         > ./format-number 7 --binary
1                                    111
> ./format-number 2 --binary         > ./format-number 8 --binary
10                                   1000
> ./format-number 3 --binary         > ./format-number 9 --binary
11                                   1001
> ./format-number 4 --binary         > ./format-number 10 --binary
100                                  1010
> ./format-number 5 --binary         > ./format-number 11 --binary
101                                  1011

We depict these values in fixed amounts such as 32 bits to discern when the line is inactive and when it is representing zeros.

All formats covered here will have larger-to-smallest for numbers. For example 100 > 010 > 001

We can represent 2^32^ in 32 bits (4 bytes). Four billion values

We can format or print a u32 value (into a String) through the following.

We calculate how many zero or one characters it takes to represent a number by taking counting leading zeros (equivalent to log2). Using std::cmp::max, we choose the length to be one if the value is value=0 so we print 0 (rather than an empty string).
We then test each bit with 0b1 & (value << i)
If the result of the bitwise and is zero (no match) then we append the 0 character else 1.

pub fn to_binary(value: usize) -> String {
    let mut buf = String::new();
    let start = std::cmp::max(usize::BITS - value.leading_zeros(), 1);
    for i in (0..start).rev() {
        let bit_set = value & (0b1 << i);
        buf.push(if bit_set == 0 { '0' } else { '1' });
    }
    buf
}
pub fn to_binary(value: usize) -> String {
    let mut buf = String::new();
    let start = std::cmp::max(usize::BITS - value.leading_zeros(), 1);
    for i in (0..start).rev() {
        let bit_set = value & (0b1 << i);
        buf.push(if bit_set == 0 { '0' } else { '1' });
    }
    buf
}

There are a few other ways of doing this, but this seemed the neatest

Parsing is the reverse of this, we take each character in the string, checking if it is 0 and add the result (through bitwise or). Each iteration we shift the accumulator value one bit to the left (equivalent to multiplying by two).

pub fn parse_binary(on: &str) -> Result<usize, ()> {
    let mut value = 0usize;
    for byte in on.bytes() {
        value <<= 1;
        if let b'0' | b'1' = byte {
            value |= (byte == b'1') as usize;
        } else {
            return Err(());
        }
    }
    Ok(value)
}
pub fn parse_binary(on: &str) -> Result<usize, ()> {
    let mut value = 0usize;
    for byte in on.bytes() {
        value <<= 1;
        if let b'0' | b'1' = byte {
            value |= (byte == b'1') as usize;
        } else {
            return Err(());
        }
    }
    Ok(value)
}

Moving on, in the real world we use base 10 or denary. Likely because most of the population has ten fingers.

The logic is similar but because 10 is not a power of 2 we use division instead of shifts and modulo instead of bitwise and for testing bits.

pub fn to_denary(value: usize, separator: &str) -> String {
    if value == 0 {
        return "0".to_owned();
    }
    let mut buf = String::new();
    for i in (0..=value.ilog10()).rev() {
        let j = (value / 10i32.pow(i) as usize) % 10;
        buf.push(b"0123456789"[j] as char);

        if i > 0 && i % 3 == 0 {
            buf.push_str(separator);
        }
    }
    buf
}
pub fn to_denary(value: usize, separator: &str) -> String {
    if value == 0 {
        return "0".to_owned();
    }
    let mut buf = String::new();
    for i in (0..=value.ilog10()).rev() {
        let j = (value / 10i32.pow(i) as usize) % 10;
        buf.push(b"0123456789"[j] as char);

        if i > 0 && i % 3 == 0 {
            buf.push_str(separator);
        }
    }
    buf
}

Here this function also takes a separator paramter. This allows formatting 123456789 to return in more readable 123 456 789 form when separator=" ".

Parsing is similar. Again because 10 is not a power of 2 we use multiplication instead of shifts and addition rather than bitwise ors.

pub fn parse_denary(on: &str) -> Result<usize, ()> {
    let mut value = 0usize;
    for byte in on.bytes() {
        value *= 10;
        if let b'0'..=b'9' = byte {
            value += usize::from(byte - b'0')
        } else {
            return Err(());
        }
    }
    Ok(value)
}
pub fn parse_denary(on: &str) -> Result<usize, ()> {
    let mut value = 0usize;
    for byte in on.bytes() {
        value *= 10;
        if let b'0'..=b'9' = byte {
            value += usize::from(byte - b'0')
        } else {
            return Err(());
        }
    }
    Ok(value)
}

Other radix are also commonly used such as base 8, 16 and 64. These are respectively named octal, hexadecimal and base64 (it needs a better new name).

These use similar logic to that of binary (as they are powers of 2). Hex has an extended alphabet beyond the 9 character introducing ABCDEF. Additionally, these formats formatting and parsing logic inspects more than one bit at a time during formatting.

Natural language

Moving past symbols, we can use words to spell out numbers. As we will see it isn’t as simple as printing 135 as one three five.

We have various scales: digits (1), tens (10), hundreds (100) and thousands (1000). Larger values then break into combinations with tens of thousands (10 000) and hundreds of thousands (100 000). When we get to thousands and thousands it starts to become confusing, so we break into a new term: the millions.

Thousands of millions is billions and millions of thousands of billions is trillions. We notice that millions has a m prefix and for billions we are at the 2 prefix that is used from binary. Successive quantaties are trillions, quadrillions.

In the following, we see that a number formatted to English roughy follows a form where we have a list of triples. A term between 1-999 followed by a magnitude. There is no magnitude for one, so we leave it out.

In English we have names for 0-10 and additionally 11-13. 14-20 is + "teen". twenty has tw-o prefix. and so on. All have names

fn under_100_to_str(value: usize) -> &'static str {
  match value {
    0 => "",
    1 => "one",
    2 => "two",
    3 => "three",
    4 => "four",
    5 => "five",
    6 => "six",
    7 => "seven",
    8 => "eight",
    9 => "nine",
    10 => "ten",
    11 => "eleven",
    12 => "twelve",
    13 => "thirteen",
    14 => "fourteen",
    15 => "fifthteen",
    16 => "sixteen",
    17 => "seventeen",
    18 => "eighteen",
    19 => "nineteen",
    20 => "twenty",
    30 => "thirty",
    40 => "fourty",
    50 => "fifty",
    60 => "sixty",
    70 => "seventy",
    80 => "eighty",
    90 => "ninety",
    number => unreachable!("{number}"),
  }
}
fn under_100_to_str(value: usize) -> &'static str {
  match value {
    0 => "",
    1 => "one",
    2 => "two",
    3 => "three",
    4 => "four",
    5 => "five",
    6 => "six",
    7 => "seven",
    8 => "eight",
    9 => "nine",
    10 => "ten",
    11 => "eleven",
    12 => "twelve",
    13 => "thirteen",
    14 => "fourteen",
    15 => "fifthteen",
    16 => "sixteen",
    17 => "seventeen",
    18 => "eighteen",
    19 => "nineteen",
    20 => "twenty",
    30 => "thirty",
    40 => "fourty",
    50 => "fifty",
    60 => "sixty",
    70 => "seventy",
    80 => "eighty",
    90 => "ninety",
    number => unreachable!("{number}"),
  }
}

Interestingly, in French 80 is quatre vingts which translates to four twenties ().

Now we have an idea about the composition, we can start working on formatting and printing. We split the input into triples with prefixes of 0-999, print this as the prefix and then append a magnitude (thousands, millions etc)

We have a few conditions for edge cases such as not printing "one million, zero thousand and zero".

pub fn to_english(value: usize) -> String {
    fn format_part(part: usize, is_last: bool, buf: &mut String) {
        let hundred = part / 100;
        if hundred != 0 {
            if !buf.is_empty() {
                buf.push_str(", ");
            }
            buf.push_str(under_100_to_str(hundred));
            buf.push_str(" hundred");
        }
        let part = part % 100;
        if part != 0 {
            if !buf.is_empty() {
                if is_last || hundred != 0 {
                    buf.push_str(" and ");
                } else {
                    buf.push_str(", ");
                }
            }
            if part <= 20 {
                buf.push_str(under_100_to_str(part));
            } else {
                let digit = part % 10;
                let tens = part - digit;
                buf.push_str(under_100_to_str(tens));
                if digit != 0 {
                    buf.push(' ');
                    buf.push_str(under_100_to_str(digit));
                }
            }
        }
    }

    const PART_WIDTH: u32 = 3;
    const MAGNITUDES: &[&str] = &[
        "",
        " thousand",
        " million",
        " billion",
        " trillion",
        " quadrillion",
        " quintillion",
    ];

    if value == 0 {
        return "zero".to_owned();
    }

    let mut buf = String::new();
    for i in (0..=value.ilog10() / PART_WIDTH).rev() {
        let size = 10_usize.pow(i * PART_WIDTH);
        let left = value / size;
        let part = left % 1000;
        let after = value - (left * size);
        let is_last = after == 0;
        format_part(part, is_last, &mut buf);
        if !buf.is_empty() && part != 0 {
            buf.push_str(MAGNITUDES[i as usize]);
        }
    }
    buf
}
pub fn to_english(value: usize) -> String {
    fn format_part(part: usize, is_last: bool, buf: &mut String) {
        let hundred = part / 100;
        if hundred != 0 {
            if !buf.is_empty() {
                buf.push_str(", ");
            }
            buf.push_str(under_100_to_str(hundred));
            buf.push_str(" hundred");
        }
        let part = part % 100;
        if part != 0 {
            if !buf.is_empty() {
                if is_last || hundred != 0 {
                    buf.push_str(" and ");
                } else {
                    buf.push_str(", ");
                }
            }
            if part <= 20 {
                buf.push_str(under_100_to_str(part));
            } else {
                let digit = part % 10;
                let tens = part - digit;
                buf.push_str(under_100_to_str(tens));
                if digit != 0 {
                    buf.push(' ');
                    buf.push_str(under_100_to_str(digit));
                }
            }
        }
    }

    const PART_WIDTH: u32 = 3;
    const MAGNITUDES: &[&str] = &[
        "",
        " thousand",
        " million",
        " billion",
        " trillion",
        " quadrillion",
        " quintillion",
    ];

    if value == 0 {
        return "zero".to_owned();
    }

    let mut buf = String::new();
    for i in (0..=value.ilog10() / PART_WIDTH).rev() {
        let size = 10_usize.pow(i * PART_WIDTH);
        let left = value / size;
        let part = left % 1000;
        let after = value - (left * size);
        let is_last = after == 0;
        format_part(part, is_last, &mut buf);
        if !buf.is_empty() && part != 0 {
            buf.push_str(MAGNITUDES[i as usize]);
        }
    }
    buf
}

We do similar for parsing, we break the input string into parts (where each part lines up with the triple structure) and do reverse lookup. The code is a bit longer so is skipped for this post.

Notenote and refers to addition and each item prefixes a magnitude (with a lower precedence) which is multipied by the magnitude.

Roman numerals

On to the title and raison d'etre for this post. Roman numerals are an ancient form of denoting amounts.

The crop up in analog clock faces, television production dates and enumerating royals with the same forename. The traditional dialect is limited to representing the numbers between 1-3999.

Their design uses several uppercase characters. I=1 being the smallest, then V=5, X=10, L=50, C=100, D=500 and finally M=1000. We see that all digits take the form of or .

Similar to the above systems we sum each part, so DLV is 500 + 50 + 5 = 555.

However, we can prefix digits to make them smaller. We refer to 9 as IX and 14 as XIV. The prefix can only precede digits up to then next so 99 is XCIX (XC + IX = 90 + 9) and not IC.

Formatting is similar to denary. Then the logic picks a certain sequence for each character

pub fn to_roman_numeral(value: usize) -> String {
    let mut buf = String::new();
    if value == 0 {
        return String::default();
    }

    assert!(value < 4000, "cannot format after 3999");

    const PARTS: &[(&str, &str, &str, &str)] = &[
        // ("III", "IIII", "VIII", "IX"), if clock
        ("III", "IV", "VIII", "IX"),
        ("XXX", "XL", "LXXX", "XC"),
        ("CCC", "CD", "DCCC", "CM"),
        ("MMM", "", "", ""),
    ];

    for i in (0..=value.ilog10()).rev() {
        let digit = (value / 10_usize.pow(i)) % 10;
        let (zero_to_three, four, five_to_eight, nine) = PARTS[i as usize];
        let value = match digit {
            0..4 => &zero_to_three[..digit],
            4 => four,
            5..9 => &five_to_eight[..digit - 4],
            9 => nine,
            value => unreachable!("{value}"),
        };
        buf.push_str(value);
    }
    buf
}
pub fn to_roman_numeral(value: usize) -> String {
    let mut buf = String::new();
    if value == 0 {
        return String::default();
    }

    assert!(value < 4000, "cannot format after 3999");

    const PARTS: &[(&str, &str, &str, &str)] = &[
        // ("III", "IIII", "VIII", "IX"), if clock
        ("III", "IV", "VIII", "IX"),
        ("XXX", "XL", "LXXX", "XC"),
        ("CCC", "CD", "DCCC", "CM"),
        ("MMM", "", "", ""),
    ];

    for i in (0..=value.ilog10()).rev() {
        let digit = (value / 10_usize.pow(i)) % 10;
        let (zero_to_three, four, five_to_eight, nine) = PARTS[i as usize];
        let value = match digit {
            0..4 => &zero_to_three[..digit],
            4 => four,
            5..9 => &five_to_eight[..digit - 4],
            9 => nine,
            value => unreachable!("{value}"),
        };
        buf.push_str(value);
    }
    buf
}

Now we can generate them, we can perform the inverse and parse them.

When parsing a roman numeral, we first turn each and item into its respective character and then process each letter in the sequence either by

Incrementing the accumulator by some amount
Decrement some future (larger) letter

And here is where the neat two line of code snippet I landed on comes in

acc += letter;
acc -= 2 * (acc % letter);
acc += letter;
acc -= 2 * (acc % letter);

We increment the value as standard, then if the letter leaves a remainder with acc we subtract two of the remainder from the accumular.

For example for IX, after then I we are left with 1 in the accumulator. We add the value of X, leaving 11 in the accumulator. We then find 11 % 10 is 1. We subtract 2 * 1 from 11 arriving at 9!

This isn't a true a roman numeral parsing, it supports a superset. It does not check that only one prefix is allowed and the prefix needs to be one less order of ten than the rest. This allows non-unique identifiers as ./parse-number IIX --roman and ./parse-number VIII --roman both return 8. A future improvement would be to add more logic to enforce the rule if required.

Try it out

You can download the command line binaries: parse-number and format-number to try it out for yourself.

Testing and benchmarking (Bonus)

Now we have our library we want to make sure that the logic is not complete baloney.

We will employ some techniques that a contributor showed me when adding fuzzing to one of my projects.

The first is the concept of a round trip. As formatting and parsing are inverse operations, we can tests that number = parse(format(number)).

As Roman numerals are limited to representing the 1-4000 we can just test them individually. We also have to test that it doesn't just print and parse zero each time so we will add a few unit tests as well.

However, for other formats we want to test a much larger range. While we could test for i in 0..=4_000_000_000 this takes a lot of compute and slows down our testing cycle (especially if we are test each representation). Instead, we will take a large sample of numbers by picking random numbers in this range.

We do not require the results to be secure so we will use the Mulberry32 generator with an initial seed by the system time and we will run the generator one hundred thousand times.

let elapsed = SystemTime::now()
    .duration_since(SystemTime::UNIX_EPOCH)
    .unwrap();

let value = (elapsed.as_micros() % 1000) as u32;
let mut random = Mulberry32 { seed: value };

for _ in 0..100_000 {
    let random_value = random.next_value();
    let pow: u32 = 10_u32.pow((random_value.rotate_left(16) % 8) + 1);
    let value = (random_value % pow) as usize;

    let value_english = to_english(value);
    let out_value = parse_english(&value_english).unwrap();

    assert_eq!(value, out_value, "{value_english} parsed as {out_value}");
}
let elapsed = SystemTime::now()
    .duration_since(SystemTime::UNIX_EPOCH)
    .unwrap();

let value = (elapsed.as_micros() % 1000) as u32;
let mut random = Mulberry32 { seed: value };

for _ in 0..100_000 {
    let random_value = random.next_value();
    let pow: u32 = 10_u32.pow((random_value.rotate_left(16) % 8) + 1);
    let value = (random_value % pow) as usize;

    let value_english = to_english(value);
    let out_value = parse_english(&value_english).unwrap();

    assert_eq!(value, out_value, "{value_english} parsed as {out_value}");
}

We also randomise the magnitude of the values, because with ranges in the billions it is a one in a million that we get a number <1000 and we want a few of those to be tested in the parse to make sure parsing with a single part works.

See the full testing here

Benchmarking

This random sample also gives a good dataset of values to time the checker on. Splitting the logic for testing English values up and wrapping each with std::time::Instant::now and elapsed(), we can see that for 100 000 numbers

format 37.444574ms. parse 89.915595ms
format 37.444574ms. parse 89.915595ms

In the future we could in benchmark different bases and experiment with SIMD for parsing values

Conclusion

Hopefully this was an interesting post covering number representations and logic for formatting and parsing them.

You can use the functions in your projects through the crate on crates.io here and you can view more of the code in its (temporary) home here.

More blog posts to come. I wrote down outlines for 25 blog posts last week then wrote this (#XXVI) instead lol

Ezno 2025

2025-07-03T00:00:00Z

It has been over 6 months since the last post and after a break I have a little bit of time again to work on my type checker and compiler project!

Firstly some random achievements from last year (2024) that I have not got around to sharing yet.

Not-so-recent achievements

Adding narrowing

Narrowing is the modification of the "backing" type of parameters and variables based on runtime expressions. It is now implemented in the checker.

This marks the last big feature of the TypeScript type system to be implemented. Remaining blockers in the type checker are either related to synthesis across modules, async and generator code and the finishing of my other crazy type system experiments.

If there is interest I may do a dedicated post on the process of implementing type narrowing.

Support for `Proxy`

My arch-nemesis in the JavaScript language, Proxy is a special object that can have custom behaviour when used. Basic partial application and some subtyping checks (but not all) are implemented in the checker.

Additionally thanks to a PR, support for RegExps also works. At some point I write a blog summarising how custom objects such as Proxy and RegExp work in the checker.

Tree-shaking

While the parser and type-checker make up most of the code in the project, there are other tools that use the generated AST and type checking information. Thanks to behavior in the checker, it does not become too difficult to add tree-shaking to the output. Because of support for the Proxy above, the following becomes possible.

While this is based on results from the type checker, it is not the actual types that enable this behaviour.

New lexer design

A parser turns strings into a tree-like data structure representing the program. Typically this is done by first "tokenising" the input string and then passing that off to a recursive-descent-parser to construct the data structure.

Previously I got a bit carried away with an over-engineered design that produced tokens in one thread and read them in a second. This had lots of problems with edge case syntax where the lexer required information from the higher level parsing.

In an on-going effort to take learnings to make the project simpler, I removed this system in terms of a more basic implementation wrapper for a string reader.

...and often simplification leads to improved performance.

this has been merged but is not yet released under the parser crate at the time of writing

More on the parser

On the topic of the parser I have been experimenting with more features including

...and partial syntax, which enables the checker to still function while editing the source.

In the future there should be variable name suggestions at the cursor

Again if there is interest I may do a dedicated blog post on the whole of the parser

Immediately next up

Here are some things I have started but have not completed yet

Updating the specification

"The specification" is a markdown document that contains a bunch of tests for type-checking behavior.

This document is invaluable for describing the state of the project as avoiding regressions with changes to the codebase.

However, I am not satisfied with the current state. So I have started to work on updating it. The updates are as follows

Again I may write a blog post on my approach to testing the checker and some ideas I have for this tool!

Add a flag to disable the events and partial application system

The events system is a unique part of the type checker and has major consequences for checking behavior.

While it has solved some of the hard problems I have wanted to solve with this checker, it is unfortuantly quite complex and current falls short to be effective in real world projects.

Because the system does not currently work in many scenarios, it is holding back the other type checker features from being used.

So last year I started on moving the system behing a flag. With some more testing a opt in --basic flag would disable events on the project and the checker would use a simpler variable and object system that is more on par with TypeScript.

I have been wanting to do a blog post on the state of this partial implemented partial application feature but I think for now it would better as a talk. If you run a meetup and interested in this then get in contact!

Finish the inference addition

Last year I had spent some time on adding a constraint inference implementation. A feature that the Hegel type checker had.

I have a few things working but there is still more to do. It would also be good to work with some real world untyped code in parralel with specific cases.

I ended up binning the original implementation (from the announcement) as it did a lot of mutation that breaks thing in conditional branches and elsewhere. The newer implementation is now to collect requirements at the end of scope, rather than modifying the type of parameters as the parameter is being used.

Add test262 checking

For JavaScript there exists a test suite of JavaScript code. It is mainly designed for engines that evaluate the code. But we can use the over 50 000 tests cases for the parser in this project.

On my WIP branch, the parser passes 91.8% of these. I should note that of the 4111 failing tests, 2395 are "parsed when should have failed". They could be duplicate parameter names, regular expressions or others. The first two are implemented at the type checker level, but could be up the chain under a flag. Now that this statistic is tracked it should only go upwards!

While the current tests only run parsing, the type checker could also be run on the files. This could uncover any crashes or places where it might not terminate. Some of these tests could also be used to test the partial application system!

An API for the type system

I think the most developed part of this project is the subtyping and disjoint code. You can see an example of how it is used here.

As that functionality is pretty much ready, it could be exposed to other project through an API.

let mut space = Space::new();

assert!(space.is_assignable_to("number", "number | string"));
assert!(!space.is_assignable_to("number | string", "number"));
let mut space = Space::new();

assert!(space.is_assignable_to("number", "number | string"));
assert!(!space.is_assignable_to("number | string", "number"));

If you are interested in using subtype logic, you can leave feedback for developing an API for it here.

Future

Other ideas that are on backburner

Continue on improving performance and architecture

While the tool is fast enough, it is a personal endeavour to learn new ways to make it even faster.

I have been working on a few things

Investigating the size of types. I have created a parser for type size information from the Rust compiler and the hope is to add some more analysis to the output to aid in the changes needed to shrink certain enums.
Performance mainly comes from structure of the data and large rewrites can be cumbersome. I have given some transformation tools a spin but I have found them too general, so I am building my own tool using tree-sitter. The hope is that it can make changes to enum and struct definitions and change corresponding usage in the parsing, to-string, checking and transformation functions.
To see whether changes have some effect I have been building on GitHub actions to track changes and run benchmarks. This includes comparing existing binaries as well as memory and CPU instructions tracking, which gives an accurate picture of performance changes.

More complete-ness: `async`, `Promise`s, reading `.d.ts` files etc and focus more on examples

Now that narrowing has been finished, the last parts of checking are async code, generators and the whole module system.

I have been pushing back some of these later features for a while. With a mode that disables the events system, it could become alot easier to get basic support working for these final items.

Releasing syntax ideas (under flags)

I have had some ideas for syntax extensions. They have been sitting in my private experiments folder but maybe it is worth posting them, less as a feature to go out and use but as some interesting ideas of what can be done with types!

Other information

Soon I will be even more available, I am looking for short-term/internship opportunities!

I have experience with: Rust, parsers/compilers, things that involve "types" and web (frontend/backend)
Random other current interests: natural language processing, internal tooling, databases and video

And again if there is a place that would be good to host a talk on the progress on the events system (or any other Rust and TypeScript stuff)! You can contact me here.

Thanks for reading the update! and all going well, more features soon!

Mapped types

2024-11-27T00:00:00Z

At the start of this year, I started implementing checking for TypeScript's mapped type feature. I always had it in the back of my mind that this would be some of the hardest logic in TSC to re-implement. While I could have just skipped types using this mapped feature and treated them as any, I instead thought that having worked on some more realistic subtype and typed things, it would be an interesting challenge to get this complicated system verified by the Ezno type checker.

For those who haven't heard of it before, the Ezno type checker is a work-in-progress but fast and correct TypeScript type checker and compiler with additional experiments.

This post covers what mapped types are, a deep dive on generic checking and representing object types and finishes up covering some advanced property features that the type checker covers. You can read see an overview the additions in the specification.

The Ezno type checker is built from the ground up. "Implementing" means getting mostly the same behaviour/output. No code or logic is copied from TSC. This post covers my approach to implementing mapped types, which may be different (or the same) as the TSC implementation. Infact there are many different behaviors in Ezno compared to TSC.

Much like many of the things in checker, there are still edge cases and a few other things that don't quite work at the moment. If you find them, then feel free to raise an issue.

The basics of object properties

In TypeScript, we can define objects with fixed keys

type Event = {
	id: string,
	name: string,
	time: Date
}

function formatDate(date: Date): string { /* ... */ }

function renderEvent(event: Event) {
	const url = `/events/${event.identifer}`;
	const formattedDate = formatDate(event.name);
	return `<a href="${url}">
		${event.name} @ ${formattedDate}
	</a>`;
}
type Event = {
	id: string,
	name: string,
	time: Date
}

function formatDate(date: Date): string { /* ... */ }

function renderEvent(event: Event) {
	const url = `/events/${event.identifer}`;
	const formattedDate = formatDate(event.name);
	return `<a href="${url}">
		${event.name} @ ${formattedDate}
	</a>`;
}

Here we define a Event type with an object. We represent properties as a list of key-value pair tuples.

properties = [
	("id",   *string*),
	("name", *string*),
	("date", *Date*  ),
]
properties = [
	("id",   *string*),
	("name", *string*),
	("date", *Date*  ),
]

This representation of events is greatly simplified for now but will be built on later.

There are several parts of the renderEvent function where we have to inspect the properties of Event.

First we see event.identifer. With this access expression we want to attempt to get the value associated with the "identifier" property key from the property list above. Finding the associate property value is quite straightforward to do in Rust using iterators.

let want_key = "identifier";
let value = properties
	.iter()
	.rev()
	.find_map(|(key, value)| 
		(key == want_key).then_some(value)
	);
let want_key = "identifier";
let value = properties
	.iter()
	.rev()
	.find_map(|(key, value)| 
		(key == want_key).then_some(value)
	);

However, when we run the above, we do not find any property pair whose key (the first item in the pair) is "identifier". From this we raise a type error.

We also return any here, so we can continue type checking.

Later on in the function we see event.name passed as an argument to formatDate. Again we run our search and this time it finds a key == "time". The RHS value in this pair is the string type. As it used as argument we check it against the date: Date parameter of formatDate and therefore find another error as we cannot assign string to Date.

Raising these type errors hints the programmer at how to correct the code. In this fixed version we can correct the access to use an actual property.

function renderEvent(event: Event) {
	const url = `/events/${event.id}`;
	const formattedDate = formatDate(event.time);
	return `<a href="${url}">
		${event.name} @ ${formattedDate}
	</a>`;
}
function renderEvent(event: Event) {
	const url = `/events/${event.id}`;
	const formattedDate = formatDate(event.time);
	return `<a href="${url}">
		${event.name} @ ${formattedDate}
	</a>`;
}

More advanced shapes of objects

This first example is relatively simple. However, keys can get more complex than this. For example the Array type has the number type as a property key.

interface Array<T> {
	[number]: T,
	// ...
	push(t: T): number,
	// ...
}
interface Array<T> {
	[number]: T,
	// ...
	push(t: T): number,
	// ...
}

There are some complications this brings

This key is not singleton: many keys match to this property value T. e.g. We want x[0], x[5] etc to both yield T
This key is not a string: we have to do a bit analysis to see if the key matches the number type. === does not cut the mustard
Additionally we want to keep existing string key behaviour on the same type. Will still want accesses such as x["push"] to yield actual methods on the Array type.

In JavaScript all keys are strings (or symbols). When we have a type here, it either depicts some generic, a string or something that coerces simple to a string (such as number and boolean).

To support types as properties we need to expand the variants of key beyond a String.

- type PropertyKey = String;
+ enum PropertyKey<'a> {
+     Constant(Cow<'a, str>),
+     Type(TypeId)
+ }
- type PropertyKey = String;
+ enum PropertyKey<'a> {
+     Constant(Cow<'a, str>),
+     Type(TypeId)
+ }

While we could have PropertyKey = Constant(String) | NumberLike. We will use the TypeId as it will handle more complex scenarios when we move on to the next part and further into mapped types.

We use Cow<'a, str> because we use the PropertyKey type in two places, one for storing properties where it 'static (owned) and one for lookup where it uses an existing &'a str slice from the AST. Using Cow<'a, str> instead of String makes it simple to reuse the definitions for both storage and lookup without cloning bytes unnecessarily (if we had a string member, we would have to .to_owned() on AST just to perform a byte equality).

Generic types with generic property keys

Now that property key can point to a Type (TypeId is a pointer to a Type) we can have a look at implementing the following Record helper type. A Record is object type who has keys of the type of generic K with values in generic V.

type Record<K extends string, V> = { [P in K]: V }
type Record<K extends string, V> = { [P in K]: V }

Here the Record type is represented in the system as a type alias with two generic parameters and a single properties with the pairing (PropertyKey::Type(*K*), *V*). In this pseudo-representation.

Record *100* = Type::Alias {
	name = "Record",
	parameters = Some([ 
		// 5 is type id string
		("K" extends *5*, id=*101*), 
		// 1 is type id any/unknown. 
		("V" extends *1*, id=*102*) 
	])
	value = Type::AnonomousObjectLiteral(
		[
			(
				// key of type K
				PropertyKey::Type(*101*), 
				// value of type V
				*102*,
			)
		]
	)
}
Record *100* = Type::Alias {
	name = "Record",
	parameters = Some([ 
		// 5 is type id string
		("K" extends *5*, id=*101*), 
		// 1 is type id any/unknown. 
		("V" extends *1*, id=*102*) 
	])
	value = Type::AnonomousObjectLiteral(
		[
			(
				// key of type K
				PropertyKey::Type(*101*), 
				// value of type V
				*102*,
			)
		]
	)
}

Now with our type we can instate it with some specific types. Let's say we are getting some data from an API which returns JSON. Once we have parsed the JSON we know from some assumptions that the results are number. So we have a Record<string, number>. We represent this using a partially applied generic.

Record<string, number> = Type::PartiallyAppliedGeneric {
	on: *100* // Record type without generic arguments
	arguments: [
		(*101* = *5*), // K = string
		(*102* = *4*), // V = number
	]
}
Record<string, number> = Type::PartiallyAppliedGeneric {
	on: *100* // Record type without generic arguments
	arguments: [
		(*101* = *5*), // K = string
		(*102* = *4*), // V = number
	]
}

A Map would be a better data structure to use here, but we are doing working in the bounds of JavaScript with the untransformed output of JSON.parse.

During instantiation we also check that the first type argument string extends string (obviously holds as subtyping is a reflexive relation)

Using the partially applied generic, we don't create a whole new object and substitute values, we just pair the Record template with some values.

This step is exactly the same as regular generic objects, nothing new for mapped types just yet.

Recording generics with the generic chain

Before we can understand how we can access property keys that are generic types, will take a short deep dive into how reading the wrapped generics works.

We know that Box<number> is a pairing of the Box type with a map of type arguments to type parameters (e.g. Box<number> = (Box, T = number)). When we get a property from this type, we split off the generic arguments and store them in a table. We then look for the property on the naked generic type. If we find a value here we return it with the current chain table. For the following

interface Box<T> { inner: T }

function getInner(box: Box<number>) {
	return box.inner
}
interface Box<T> { inner: T }

function getInner(box: Box<number>) {
	return box.inner
}

The process is something like

We start by requesting the "inner" property on Box<number>
We unfold the type arguments and so we now look for "inner" on Box with substitutions: [(T, number)].
In this example "inner" does exist on Box so we return Logical::Implies { result: PropertyValue::Value(T), substitutions } where we also send our substitutions chain with the result
After lookup we found a valid result T. We find there are some substitutions we scan T and follow any lookups from the substitutions: [(T, number)] list.
We find a T can be substituted with number and so return number

We return a Logical::Implies adjoined with substitutions in a middle step because it works better in Rust's borrow checking. While in this case the substitution was a simple swap, in other cases we need to register new types and is easiest to create new types (mutate the store) only after the traversal of current types has finished.

Subtyping with generics

Above we used the GenericChain for lookup. We also use this structure for subtyping of generic items.

interface Box<T> { inner: T }

const numbers: Box<number> = { inner: "not number" };
interface Box<T> { inner: T }

const numbers: Box<number> = { inner: "not number" };

When checking (subtyping) the following happens with the table. (each line somewhat represents call to the is_subtype function)

Box<number>                            :> { inner: "not number" }

// Unfold the `number` argument
{ inner: T }         (with T = number) :> { inner: "not number" }

// iterating over properties on the LHS. We only have `inner` property
{ inner: T }.inner   (with T = number) :> { inner: "not number" }.inner
T                    (with T = number) :> "not number"
// Replace T with number here using this chain
number               (with T = number) :> "not number" // <- subtyping returns here
Box<number>                            :> { inner: "not number" }

// Unfold the `number` argument
{ inner: T }         (with T = number) :> { inner: "not number" }

// iterating over properties on the LHS. We only have `inner` property
{ inner: T }.inner   (with T = number) :> { inner: "not number" }.inner
T                    (with T = number) :> "not number"
// Replace T with number here using this chain
number               (with T = number) :> "not number" // <- subtyping returns here

In the above we see that the LHS unfolds its generics and so we can reduce them out to checking on the leaves.

Note the RHS can also produce a table if it is generic.

In the above example I simplified substitutions to just be a list. In the actual code this is called GenericChain and is a linked list of substitution lists. This allows it to do nested generics. So the following is valid.

interface Box<T> {
	parent: Box<number> | null
	inner: T
}

const myBox: Box<string> = { parent: { parent: null, inner: 6 }, inner: "heya" }
interface Box<T> {
	parent: Box<number> | null
	inner: T
}

const myBox: Box<string> = { parent: { parent: null, inner: 6 }, inner: "heya" }

// Checking
Box<string>                            :> { ... }
{ ... } with (T = string)

{ ... } with (T = string) (T = number) :> { ... }
T       with (T = string) (T = number) :> 6
// We pick the first from the right
number                                 :> 6
// Checking
Box<string>                            :> { ... }
{ ... } with (T = string)

{ ... } with (T = string) (T = number) :> { ... }
T       with (T = string) (T = number) :> 6
// We pick the first from the right
number                                 :> 6

Additionally GenericChain uses lifetimes of arguments rather than cloning lists (and thus allocating).

Testing generic keys

Given we now have these generics basics out of the way, we have to define what these type representations contain.

It starts to get more complicated for type keys. One case of this is the Array type. As previously stated the Array type has a numeric property and for this we can't really use equality. :/

PropertyKey::Type(TypeId::NUMBER_TYPE) 
  == PropertyKey::String(Cow::Borrowed("1"))
PropertyKey::Type(TypeId::NUMBER_TYPE) 
  == PropertyKey::String(Cow::Borrowed("1"))

So instead we need to make a special function for key validity that we can put into the lookup function.

This is where the complications start. While we could use type_is_subtype we don't have two TypeIds as we sometimes have a PropertyKey::String on a side. We also have a problem in the above example. "1" is string not a number, so we have to add catch for the implicit conversion JavaScript when assigning and reading properties with keys that are numbers.

Unfortunately we do have a have some duplication of some logic but we have a function like

fn key_matches(
	base: PropertyKey,
	want: PropertyKey,
) -> bool {
	// simple case
	if (
		PropertyKey::String(base), 
		PropertyKey::String(want))
	) = (base, want) {
		return base == want
	}

	// catch for number
	if let (
		PropertyKey::Type(TypeId::NUMBER_TYPE), 
		PropertyKey::String(want)
	) = (base, want) {
		return want.parse::<usize>().is_ok();
	}

	// delegate to regular subtyping
	if let (
		PropertyKey::Type(base), 
		PropertyKey::Type(want)
	) = (base, want) {
		return type_is_subtype(base, want).is_subtype();
	}
}
fn key_matches(
	base: PropertyKey,
	want: PropertyKey,
) -> bool {
	// simple case
	if (
		PropertyKey::String(base), 
		PropertyKey::String(want))
	) = (base, want) {
		return base == want
	}

	// catch for number
	if let (
		PropertyKey::Type(TypeId::NUMBER_TYPE), 
		PropertyKey::String(want)
	) = (base, want) {
		return want.parse::<usize>().is_ok();
	}

	// delegate to regular subtyping
	if let (
		PropertyKey::Type(base), 
		PropertyKey::Type(want)
	) = (base, want) {
		return type_is_subtype(base, want).is_subtype();
	}
}

Full key_matches function here

We could create PartialEq implementation but 1) it is not equality, as it involves subtyping and 2) as we will see in a minute this function will start to take a few more parameters and as we are only fixed to LHS and RHS parameters in PartialEq::eq we will we use a function instead

Passing generics through `key_matches`

In our initial Record example we have a key that is a generic parameter. With the above we have a problem as we do subtyping on K but we don't have a value for K. Our key_matches function isn't aware of type parameters.

So we modify key_matches function to allow generic arguments down. This base generics will contain a list which has the K=string pair in. When we do subtyping we find a K on the LHS, and using our list we pass down, we look up the argument for K and find string. Therefore if want is a string like property key then the key_matches function returns in this example true!

fn key_matches(
	(base, base_generics): (PropertyKey, GenericChain),
	(want, want_generics): (PropertyKey, GenericChain),
) -> bool {
	// ...

	if let (PropertyKey::Type(base), PropertyKey::Type(want)) = (base, want) {
		return type_is_subtype((base, base_generics), (want, want_generics)).is_subtype();
	}
}
fn key_matches(
	(base, base_generics): (PropertyKey, GenericChain),
	(want, want_generics): (PropertyKey, GenericChain),
) -> bool {
	// ...

	if let (PropertyKey::Type(base), PropertyKey::Type(want)) = (base, want) {
		return type_is_subtype((base, base_generics), (want, want_generics)).is_subtype();
	}
}

When this returns true we now have the following lookup. As it returns true, we yield the value and this in turn is returned with the chain so that we can substitute values in.

// From `Record<string, number>`
let base_generic_chain = *[(K, TypeId::STRING_TYPE), (V, TypeId::NUMBER_TYPE)]*;
let want_key = PropertyKey::String(Cow::Borrowed("something"));
let value = base_properties
	.iter()
	.rev()
	.find_map(|(key, value)| 
		key_matches((key, base_generic_chain), (want_key, None)).then_some(value)
	);

return value.map(|value| Result {
	 value,
	 substitutions: base_generic_chain
})
// From `Record<string, number>`
let base_generic_chain = *[(K, TypeId::STRING_TYPE), (V, TypeId::NUMBER_TYPE)]*;
let want_key = PropertyKey::String(Cow::Borrowed("something"));
let value = base_properties
	.iter()
	.rev()
	.find_map(|(key, value)| 
		key_matches((key, base_generic_chain), (want_key, None)).then_some(value)
	);

return value.map(|value| Result {
	 value,
	 substitutions: base_generic_chain
})

This is how Record<string, numbers> works. It isn't quite a mapped type yet, but hopefully you have a bit more of an understanding of generics to see how this works now.

Additionally I hope you can see why the properties are stored as a Vec here (rather than a HashMap). Not only is it a little overkill for the average < 10 key based object. There isn't a simple hash and equality relation for when keys based on types are in the picture.

On to mapped keys

Real mapped types

So far we have covered constant keys and keys that are passed from above via a generic.

(real) Mapped types are when the property value is based on the type of a property key.

We do this using the [ParameterName in ...]: Value syntax for a property. Here ParameterName is a new generic parameter (similar to that on functions or interfaces). It is can be used as a value in the Value type annotation.

For example, in the following we build a type that maps over the keys of Event and builds a new type where each field might be null. We can see it uses type P as a inline property generic. We use in to say it must extend the type keyof Event (covered more in a minute). We use this generic P in the property value to depict the result is the associate value of Event[P].

type Event = {
	id: string,
	name: string,
	time: Date
}

type EventButFieldsPossiblyNull = {
	[P in keyof Event]: Event[P] | null
}
type Event = {
	id: string,
	name: string,
	time: Date
}

type EventButFieldsPossiblyNull = {
	[P in keyof Event]: Event[P] | null
}

We have mapped each property to a new value using P as a parameter.

This practise is normally parameterised and used for utility types.

type TButFieldsPossiblyNull<T> = {
	[P in keyof T]: T[P] | null
}
type TButFieldsPossiblyNull<T> = {
	[P in keyof T]: T[P] | null
}

They are a niche feature, but like many other abstractions they can be useful when

Don't want to copy out definitions of existing items
Keeping definitions up-to date/inline within related definitions
Working on top of definitions we don't own (external or generated etc)

Again this partial applied. "creating" a mapped type does not walk the properties, it just creates a partially applied generic. This reduces overhead on instantiation, but moves complexity to usage and means that each usage has to do some work.

Important functionality: Inferring type argument values

Before talking about the access part. I will talk about a new kind kind of generic recording. We previously saw how type arguments on a structure are passed down through access and subtyping with GenericChain the table like structure.

There are cases where we have a generic but don't have a value for it. Instead we need to infer its value based on usage. This is where Contributions comes in.

For example given the function

declare function idString<T extends string>(t: T): T;
declare function idString<T extends string>(t: T): T;

When we call it with idString("Hello World") we want to find a type for this expression and this requires knowing figuring out the returned type. In this example we need to infer T to have the value "Hello World", which we will then use to substitute as the return type.

The implementation to do lies in ~~making~~ subtyping ~~more complex~~. When we call a function we subtype each argument value against its respective parameter type. In this case we run (T extends string) :> "Hello World". The inner steps are as follows

We find that the LHS T is a (substitutable) generic parameter
We find that it does not have a value in the LHS GenericChain so continue to subtype the generic (there is no GenericChain in this example).
We also do not find for this FunctionGeneric that there is a explicit call side type argument in state.call_site_type_arguments so continue to subtype the generic
For generics we check the extends constraint (if there was no extends annotation, this annotation would extend any and so this check always passes)
We find the constraint check passed ("Hello World" does indeed extend string!)
The important part: take our LHS generic type T and RHS type "Hello World" and we append T="Hello World" as a tuple pair into the contributions list in state. Something like contributions.push((lhs, rhs)).

Contributions is inside the subtyping::State structure which is passed through the is_subtype function (as well as the generic chains).

There are some fun details with contributions such as dropping cases with union types when a branch fails. We can also see that through the NoInfer intrinsic the results are dropped. This is why they are internally held as Vec because that is easy to drop ranges from as it is ordered.

By doing it this way it allows inference to be done in the same scan by the type checker. Although this if faster and simpler, it does have complications (not impossible though) for inference that uses parameters before a value to the right has been checked and inferred.

Mapped keys: The identity key case

After you have passed inference 101 above, you should be good to go for getting you head around how mapped types work.

We will start of with the identity mapped type.

type IdOnKeys = { [P in string]: P }
type IdOnKeys = { [P in string]: P }

When accessing ["a"] it goes through the steps outlined above

First as one of the side is key_matches it goes to subtype (or something like subtyping),
There is no value for P in the GenericChain (it is not passed from above and also it is marked as a MappedGeneric type) so we go to generic contributions
P in string implies P extends string and we have that "a" is a string so we continue
In this case we add P = "a" to the contributions! The argument for generic parameter P is inferred to be the type "a"

Now we have our pairing we need to make the access aware of this value. To do this we expand on our GenericChain table and pass *[(P, "a")]* to the value resolution.

One of the problems when attempting the addition of this feature is that inference/contributions is a a TypeId to TypeId paring. When accessing a property key we can have be passed PropertyKey::String which is not a type. We are a bit stuck as we cannot create a new type during subtyping as that would conflicts with the borrow checker (we would require a mutable reference to TypeStore and during subtyping we only have a immutable reference to the bank of types). So like so many problems faced with types, it turned out the checker needed a new pseudo type.

To work around allowing TypeId <- String contributions I introduced the new CovariantContribution enum to the contribution system. This structure has more variants and so is effectively TypeId | String | ... and like other things delegates later steps to creating a type.

Mapped types: the identity map case

The above shows a basic example of the mapping parameter keys. We will now step up in complexity to cover some more useful mapped types. The first is the identity map on objects. It takes a object and producing another object with the same keys mapping to the same values.

type IdentityOnObject<T> = {
	[P in keyof T]: T[P]
}
type IdentityOnObject<T> = {
	[P in keyof T]: T[P]
}

While this has no effect, we can see how we can now modify this above example to give it more use by applying type operations to the property key or value type.

type Pick<T, ToPick extends string> = {
	[P in (keyof T & ToPick)]: T[P]
}

type Omit<T, ToOmit extends string> = {
	[P in (keyof T & Not<ToOmit>)]: T[P]
}

type MakeArray<T> = {
	[P in keyof T]: Array<T[P]>
}
type Pick<T, ToPick extends string> = {
	[P in (keyof T & ToPick)]: T[P]
}

type Omit<T, ToOmit extends string> = {
	[P in (keyof T & Not<ToOmit>)]: T[P]
}

type MakeArray<T> = {
	[P in keyof T]: Array<T[P]>
}

While you can use the TypeScript Exclude type and it works in Ezno, I think it is simpler to use the new Not.

Key type modifiers: `in` and `as` syntax

Mapped types can have special syntax in the property position.

In the first example, it is quite simple to see K in T implies K extends T.

The harder code syntax is as. At first it got me scratching my head. But I figured that rewrites the key.

type PrefixKeys<T> = {
    [P in ((keyof T) & string) as `property_${P}`]: T[P];
};
type PrefixKeys<T> = {
    [P in ((keyof T) & string) as `property_${P}`]: T[P];
};

For the identity on properties we have the property key as infer P, for the identity on objects we have a key of type infer P extends keyof T. For this example we put insert P into the RHS type and so have property_${infer P extends ((keyof T) & string)}. In total this is a three line addition we already have the P type in the namespace for the RHS type synthesis.

More advanced property values

So far we have only seen properties having the value being a exact type. To discuss the last parts of mapped types we need to expand on the variants of PropertyValue.

- type PropertyValue = TypeId;
+ enum PropertyValue {
+     /// Simple
+     Value(TypeId),
+     /// from `get` and `set`
+     GetterSetter {
+         getter: Box<FunctionType>,
+         setter: Box<FunctionType>,
+     }
+ 
+     ///
+     ConditionallyExists {
+         condition: TypeId,
+         value: Box<Self>
+     }
+ 
+     ///
+     Configured {
+         writable: TypeId,
+         // ...
+         value: Box<Self>
+     }
+ }
- type PropertyValue = TypeId;
+ enum PropertyValue {
+     /// Simple
+     Value(TypeId),
+     /// from `get` and `set`
+     GetterSetter {
+         getter: Box<FunctionType>,
+         setter: Box<FunctionType>,
+     }
+ 
+     ///
+     ConditionallyExists {
+         condition: TypeId,
+         value: Box<Self>
+     }
+ 
+     ///
+     Configured {
+         writable: TypeId,
+         // ...
+         value: Box<Self>
+     }
+ }

The simple case becomes the main variant. We also introduce get and set functions in here, we handle them specially but that is for another blog post.

The two newer additions are PropertyValue::ConditionallyExists and PropertyValue::Configured.

`PropertyValue::ConditionallyExists`

A property can conditionally exist via adding the ? prefix modifier to property declaration.

interface MappedTypes {
	fun?: "YES!"
}
interface MappedTypes {
	fun?: "YES!"
}

This declaration is different to fun: "YES!" | undefined as in this case fun in obj is always true, whereas in the above example it is conditional. Here we say that condition for it existing is unknown and use the boolean type in its place.

This variant can represent actual JavaScript code

const obj = {};
if (condition) {
	obj.prop = 2;
}
// obj.prop conditionally exists (value = 2) based on `condition`
const obj = {};
if (condition) {
	obj.prop = 2;
}
// obj.prop conditionally exists (value = 2) based on `condition`

Here we see that "prop" in obj is only true when condition is truthy. When we access this property we allow retrieve it but we do union the result with undefined for the user to account for later.

`PropertyValue::Configured`

The second new PropertyValue variant is a configured wrapper. In JavaScript every property has a descriptor. The usage of the these descriptors is quite low. However they crop up in standard library items (normally causing problems).

We can (ab)use the writable value of a property to mark a property as readonly. We say that when writable is false we cannot assign to it. For example if the x property is not writable, then .x = 2 will fail.

So given a property declaration prefixed with readonly

interface Item {
	readonly id: string,
	count: number,
}
interface Item {
	readonly id: string,
	count: number,
}

we define the properties as something like

[
	("id", PropertyValue::Configured { 
		value: PropertyValue::Type(TypeId::STRING_TYPE),
		writable: TypeId::FALSE
	})
	("count", PropertyValue::Type(TypeId::NUMBER_TYPE))
]
[
	("id", PropertyValue::Configured { 
		value: PropertyValue::Type(TypeId::STRING_TYPE),
		writable: TypeId::FALSE
	})
	("count", PropertyValue::Type(TypeId::NUMBER_TYPE))
]

and do not allow writing to it

There is a bit of nuance here around TypeScript readonly and JavaScript writeable. TypeScript's readonly is a developer abstraction, which has no effect on compile time. writeable on the other hand is part of JavaScript execution and has runtime behaviour when looked up Object.getPropertyDescriptor.

Most people prefer the upfront warnings of readonly, rather than marking it as immutable. JavaScript writable is hard to do compile introspection (but not impossible as showcased later), so TypeScript added their own syntax and system to add this case.

There are others here, but we skip over them for now

Mapping Modifiers: Default behaviour

When we have a mapped key that uses extends, we have to carry over these conditional and writeable modifiers. For example given

type AllValuesString<T> = {
	[P in keyof T]: string
}
type AllValuesString<T> = {
	[P in keyof T]: string
}

Then if we have AllValuesString<{ a?: number, readonly b: boolean }> then the resulting type is actually { a?: string, readonly b: string }!

This seems incredible complicated to implement at first, but it is possible.

In the implementation in Ezno, at the mapped type definition: if the key extends a keyof ... modifier, the resultant property is modified to be based on two special parameters TypeId::WRITABLE_KEY_ARGUMENT and TypeId::NON_OPTIONAL_KEY_ARGUMENT.

let mut rhs = PropertyValue::Value(...);

if references_key_of(mapped_type.extends) {
	rhs = PropertyValue::ConditionallyExists {
		condition: TypeId::NON_OPTIONAL_KEY_ARGUMENT,
		value: Box::new(PropertyValue::Configured { 
			on: rhs, 
			writeable: TypeId::WRITABLE_KEY_ARGUMENT 
		})
	};
}
let mut rhs = PropertyValue::Value(...);

if references_key_of(mapped_type.extends) {
	rhs = PropertyValue::ConditionallyExists {
		condition: TypeId::NON_OPTIONAL_KEY_ARGUMENT,
		value: Box::new(PropertyValue::Configured { 
			on: rhs, 
			writeable: TypeId::WRITABLE_KEY_ARGUMENT 
		})
	};
}

Here the property is made conditional and configured based on two special types. We can treat them as parameters. We need to infer whether they are true or false based om keyof returns.

So the next part is to connect this up from the other side. When doing subtyping for key matching, if we test against a keyof type, two arguments are set into the inferred table against for these internal parameters TypeId::WRITABLE_KEY_ARGUMENT and TypeId::NON_OPTIONAL_KEY_ARGUMENT.

When we get a property out the other side we match these up and specialise them.

Mapped type modifiers: Property descriptor and conditionally exists

So we have the default case out of the way. We can change the mapping-ness and enforce cases using special syntax.

For optionality with ?:, we set condition: TypeId::TRUE in the conditional wrapper (rather than the special mapping parameter type)

type Partial<T> = {
	[P in keyof T]?: T[P];
};
type Partial<T> = {
	[P in keyof T]?: T[P];
};

Here because of the -?: we do not add the conditional wrapper

type Required<T> = {
	[P in keyof T]-?: T[P];
};
type Required<T> = {
	[P in keyof T]-?: T[P];
};

Here with the readonly prefix we set readonly: TypeId::TRUE in the configurable wrapper (rather than the special mapping parameter type)

type Immutable<T> = {
	readonly [P in keyof T]: T[P];
};
type Immutable<T> = {
	readonly [P in keyof T]: T[P];
};

Here because of -readonly we do not add the configurable readonly wrapper!

type Mutable<T> = {
	-readonly [P in keyof T]: T[P];
};
type Mutable<T> = {
	-readonly [P in keyof T]: T[P];
};

Full logic here

And that concludes the mapped type section. Let me know in the comments if there is any mapped type feature I have missed.

The next parts go into some really interesting and important things about properties and more types!

String intrinsics: `Uppercase`, `Lowercase`, etc

When working on mapped types, I noticed some examples used special string transforms in their as clause. We can pass constant strings to these transformer types.

const x: Uppercase<"hi"> = "HI";
const y: Lowercase<"Hello"> = "hello";
// Type error: "example" is not assignable to "EXAMPLE"
const z: Uppercase<"Example"> = "example";
const x: Uppercase<"hi"> = "HI";
const y: Lowercase<"Hello"> = "hello";
// Type error: "example" is not assignable to "EXAMPLE"
const z: Uppercase<"Example"> = "example";

There are a few things that were implemented during this step

Eagerly apply the transformation, including through union types (Uppercase<"a" | "b"> = "A" | "B").
Apply when used on a generic for a return type
Subtyping.

Similar to other intrinsics, when we encounter this we have custom logic for these situations.

If anyone has a neat way to do case insensitive string equality that is better than lhs.to_lowercase() == rhs.to_lowercase() then please send a PR 😁🙏.

Introducing my own intrinsic: `CaseInsensitive`

One example that got me confused.

type Item = {
    id: string,
    iD: number
}

type MappedItem = {
    [P in keyof Item as Uppercase<P>]: Item[P]
}
type Item = {
    id: string,
    iD: number
}

type MappedItem = {
    [P in keyof Item as Uppercase<P>]: Item[P]
}

Here we have a property of type Uppercase<infer P extends keyof Event>. We can pull out the problem here and I will give you an equivalent problem: what the return type is here?.

declare function func<T extends string>(param: Uppercase<T>) -> T;
declare function func<T extends string>(param: Uppercase<T>) -> T;

We see that we have to infer through this Uppercase boundary. For it to be valid you expect func("hi") to return all four valid values for T: "hi", "Hi", "hI", "HI". To support this I added CaseInsensitive. The result is that if a inferred pair is found inside one of these case transformations then we wrap the output in CaseInsensitive to pick up all valid cases.

After adding similar subtyping logic, this fixes the problem.

I think I had to do it this way because I am doing it via inference via key_matches whereas TypeScript eagerly evaluates mapped types and so eagerly generates these variants.

If you like intrinsics, check out the last blog post.

Switching it up, here are some other things I have been working on properties.

Functions and `Object.keys`

We will now take a peek at the crazier side of the checker. One of the biggest experimental features is the observance of side-effects.

In some linear code we can apply mutations to types as the program progresses.

const obj = {}
obj.property = 2;
obj satisfies string; // fails as 2 is not a string
const obj = {}
obj.property = 2;
obj satisfies string; // fails as 2 is not a string

This simple case is quite simple to implement. But problems occur when using functions. A huge experiment in Ezno is for functions to record special properties not representable by just types. We can see this in the following example.

function setAt(obj, at: number, value) {
	obj[at] = value
}

debug_effects_rust(setAt)
function setAt(obj, at: number, value) {
	obj[at] = value
}

debug_effects_rust(setAt)

We will skip over a lot here but jumping straight into the deep end and the front of this experiment the specification has an example which shows the application of some of the Array.prototype.push mutation information. This gives us more precise information on values after mutations.

WW also account for mutations in for ... in ... loop. Again glossing over a lot here, but the assignments in the loop get collected and when we call some functions which use this, we inspect the object that is in'd and we run each assignment for each (enumerable) property key in the object.

In fact this is also how Object.entries, Object.values and Object.assign work. We define them as

declare class Object {
    static keys(on: { [s: string]: any }): Array<string> {
        const keys: Array<string> = [];
        for (const key in on) {
            keys.push(key);
        }
        return keys
    }

    static values(on: { [s: string]: any }): Array<any> {
        const values: Array<any> = [];
        for (const key in on) {
            values.push(on[key]);
        }
        return values
    }

    static entries(on: { [s: string]: any }): Array<[string, any]> {
        const entries: Array<[string, any]> = [];
        for (const key in on) {
            entries.push([key, on[key]]);
        }
        return entries
    }
}
declare class Object {
    static keys(on: { [s: string]: any }): Array<string> {
        const keys: Array<string> = [];
        for (const key in on) {
            keys.push(key);
        }
        return keys
    }

    static values(on: { [s: string]: any }): Array<any> {
        const values: Array<any> = [];
        for (const key in on) {
            values.push(on[key]);
        }
        return values
    }

    static entries(on: { [s: string]: any }): Array<[string, any]> {
        const entries: Array<[string, any]> = [];
        for (const key in on) {
            entries.push([key, on[key]]);
        }
        return entries
    }
}

and we get the following results (at least on ezno 0.0.22, it is broken in the latest Ezno and we seem to miss out on the first iteration. Hopefully I can find some time to track down what is broken here soon)

This may seems like a lot of work but this mechanism has a lot of other benefits and I think it is the coolest part of Ezno. I will eventually share the whole system and some really interesting things I have discovered.

Complications with `Object.defineProperty`

These inferred side-effect things are cool. However for some things we can't use mutations based on JavaScript code, these being

Internal APIs (aren't possible just using JS syntax)
APIs that would be quite complex to write in terms of events (also don't make sense for all inputs). For example Math.sin.

This is similar with JavaScript runtimes. For certain things they use special native code for these functions rather than just JavaScript.

The function Object.defineProperty has this property. It isn't possible to write an alternative for this function just using [key] and =.

So rather than giving this function a body and using the events it has a different behaviour for a constant function. When Object.defineProperty is called, it instead runs this internal logic in constant_functions.rs. It isn't perfect, but it does accept sensible usage of this API.

Previously we saw that we needed a readonly annotation this value. But we can do exactly the same in JavaScript

Locking objects with `freeze`, `seal` and `preventExtensions`

The initial mapped type PR included basic support for freeze. But thanks to an external PR the logic is now extended to cover preventExtensions andseal

The current representation for properties: adding `Publicity`

Over the course of this blog post we have built up the representation of properties. The final piece is that this key-value pair is a actually a triple including Publicity (public or private), PropertyKey and PropertyValue. (However full private and public property checking is unfinished).

They are stored in a array (Vec), for example the type annotation { a: 0, b: 2 } is represented something similar to

[
	(Publicity::Public, PropertyKey::String("a"), PropertyValue::Value(TypeId -> 0)),
	(Publicity::Public, PropertyKey::String("b"), PropertyValue::Value(TypeId -> 2))
]
[
	(Publicity::Public, PropertyKey::String("a"), PropertyValue::Value(TypeId -> 0)),
	(Publicity::Public, PropertyKey::String("b"), PropertyValue::Value(TypeId -> 2))
]

As discussed the type checker in Ezno is mutation (technically fully side-effect) aware and dependent so traditional representing this as obj = type with properties doesn't quite work. Instead we say obj = *object 100* and then record properties onto each object for each context.

This is because objects can have difference properties in different contexts. There isn't a globally true list of properties for a object.

const obj = { property: 1 };
// ctx.properties = { 'obj': [["property", { value: 1 }]] }
if (condition) {
	obj.property = 2;
	// ctx.properties = { 'obj': [["property", { value: 1 }], ["property", { value: 2 }]] }
	// obj.property = 2 throughout here
} else {
	console.log("something")
	// obj.property = 1 still here
	// ctx.properties = { 'obj': [["property", { value: 1 }]] }
}
const obj = { property: 1 };
// ctx.properties = { 'obj': [["property", { value: 1 }]] }
if (condition) {
	obj.property = 2;
	// ctx.properties = { 'obj': [["property", { value: 1 }], ["property", { value: 2 }]] }
	// obj.property = 2 throughout here
} else {
	console.log("something")
	// obj.property = 1 still here
	// ctx.properties = { 'obj': [["property", { value: 1 }]] }
}

The current system also enables interface merging. However not all types store their properties on the context. Thanks to a recent change, anonymous object type annotations store their properties actually on the type.

The eagle eyed among you will notice that in the first branch, obj has two properties named property. I am still unsure whether appending rather than replacing is the best approach. It does mean that assigning a property doesn't have to check the rest and not-implemented at the moment but it means that LSP hover could instead only print up-to some assignment. This system is fine though as access accounts for this finding the first match when walking right-to-left.

Improvements to listing types

The PR also made some improvements for being able to get all properties on an object. Now number properties are sorted and added first, the enumerable property is checked and properties with both getter and setter fixed.

Additionally thanks to an external PR added a while back, when a property is not find it gives alternatives using the Levenshtein algorithm on these keys.

Future changes to properties for consideration

None of this adds anything. But there is still things that could be changed with regards to this representation of properties.

Separate out PropertyValue::Configured
- Maybe this could be held separately on the context
Adding metadata to properties
- This is needed for documentation comments in the LSP
Restructuring the LocalInformation.properties
- Maybe removing PropertyValue::Optional for { a: string, b?: string } == { a: string } & ({ b: string } | {})
Publicity item absorbed in PropertyKey ?

Hopefully you enjoyed this blog post! If you haven't seen yet there are recent blog posts on experimental types and a background on type theory.

Experimental types

2024-11-13T00:00:00Z

Over this summer, while working on increasing TypeScript compatibility in the checker and tackling more advanced type problems, I had an escapade with some experimental types! These types expand on what is currently representable in TypeScript. This blog post will introduce them, when they come about and things they catch.

For those who haven't heard of it before, the Ezno type checker is a work-in-progress but fast and correct TypeScript type checker and compiler with additional experiments.

The missing operation

Set theory is based on three operations: unions, intersections and one other. We can build a union of a type with | and intersections with & but this third operation is less common and does not exist in the TypeScript type system.

This means that the types of some expressions are hard to represent. We can see this here

function func(param: string) {
  if (param !== "") {
    console.log(param)
    //          ^^^^^
  }
}
function func(param: string) {
  if (param !== "") {
    console.log(param)
    //          ^^^^^
  }
}

Here we know that param is a string (by the parameter type annotation) but we additionally know from the conditional statement it is not an empty string "" because of the condition (param !== ""). While this state doesn't cause any type problems and treating as string is mostly fine, it does miss out on information that it has a negated member. For example if we only have the string information, the type checker doesn't raise an disjoint error for the following equalities

function func(param: string) {
  if (param !== "") {
    // ...
    return param === "" ? 2 : 1;
    //     ^^^^^^^^^^^^ <- this equality is always **false** because `param !== ""`
  }
}

// or slightly more complex via control flow
function func(param: string) {
  if (param === "") {
    return 0;
  }
  // ...
  return param === "" ? 2 : 1;
  //     ^^^^^^^^^^^^ <- this equality is always false because `param !== ""`
}
function func(param: string) {
  if (param !== "") {
    // ...
    return param === "" ? 2 : 1;
    //     ^^^^^^^^^^^^ <- this equality is always **false** because `param !== ""`
  }
}

// or slightly more complex via control flow
function func(param: string) {
  if (param === "") {
    return 0;
  }
  // ...
  return param === "" ? 2 : 1;
  //     ^^^^^^^^^^^^ <- this equality is always false because `param !== ""`
}

While these examples are short and the cases are obvious, you can see how in longer functions with more conditions and other things going on these cases are get more difficult for a user to observe.

More on reachability and narrowing in a future blog post!

So we want a new kind of type which can represent the negation of some properties or members. This type is called the `Not` type and transforms a single type.

The `Not` type

So we now want to construct a type which can wrap an existing type and negate it.

A simple way that avoids changing the checkers Type declaration is to introduce a new intrinsic type. TSC already has string intrinsic types (and NoInfer). We can define another special internal type alias intrinsic called Not that wraps another type using the type parameter, similar to how regular generic type alias types work (but without eager specialisation, because it is in its final form).

Using a intrinsic type here

Avoids new syntax
Is consistent in implementation to other intrinsic types in the checker

With our new type at hand we can represent the type of our parameter inside the condition to be `param: string & Not<"">`. (Here `""` is the empty string).

Creating a negated type through narrowing

In the checker expressions in branching structures can refine types in a process known as narrowing.

There will be more on narrowing in a future blog post but for now we can say that the function that produces narrowed facts from a expression also takes a negated parameter.

typeof x === string --> narrow(negated: false, expression: ...) => NarrowedFacts

typeof x !== string --> narrow(negated: true, expression: ...) => NarrowedFacts
                                        ^^^^
typeof x === string --> narrow(negated: false, expression: ...) => NarrowedFacts

typeof x !== string --> narrow(negated: true, expression: ...) => NarrowedFacts
                                        ^^^^

For the first case we say x: string. In the second case where negated is true, when we produce the facts we wrap them all in Not, so get x: Not<string>.

For most unions (1 | 2) we can actually filter the members and don't introduce a Not type. We only introduce this for the large unions such as number, string etc. If you have any questions about narrowing that could be covered in that post, leave them in the comments below!

Exposing negated types type to annotations

While having it picked up by narrowing and the !== operator is cool, I will also be generous and add this special type to the root context for use in your programs.

We will come onto how subtyping of this negated type works in a minute, but for now we can modify a functions parameter definition to force callers of our function to not take zero as an argument.

function divide(numerator: number, denominator: number & Not<0>) { ... }
function divide(numerator: number, denominator: number & Not<0>) { ... }

You can try this example out in the online checker playground.

Subtyping and disjoint-ness of negated types

So far everything seems quite straightforward: register our intrinsic type, map negated narrows and put it in root so you can reference and use it. So you might be expecting that the usage of it in the type system is where the complexity comes? But you would be wrong, the Not mechanism can be entirely based on existing functionality.

An example of the assignment case could be passing a string argument to a parameter typed as Not<number>. If you like set theory you might have guessed already, but subtyping a Not type is based on disjoint-ness!.

For string to be a subtype of Not<number> (string <: Not<number>) is for the number and string types to be disjoint! And this is true, there is element that satisfies the contracts of both string and number.

And we already have a implementation of disjoint checking, which is used for the equality operator === etc.

So we have covered subtyping, but lets get back to the first example

function func(param: string) {
  if (param !== "") {
    // ...
    return param === "" ? 2 : 1;
    //     ^^^^^^^^^^^^ <- this equality is always **false** because `param !== ""`
  }
}
function func(param: string) {
  if (param !== "") {
    // ...
    return param === "" ? 2 : 1;
    //     ^^^^^^^^^^^^ <- this equality is always **false** because `param !== ""`
  }
}

Here we want to test whether Not<""> is disjoint with "". And it turns out this is also a really simple operation, now in reverse: for Not<""> to be disjoint with "" is for the empty string to be a subtype of the empty string ("" <: "") which is true as subtyping is a reflexive relation (T = T => subtype).

As another example testing whether Not<string> is disjoint with "hello" is testing whether "hello" is a subtype of string. Thus if we have a strict equal operate check in TypeScript we know that (x satisfies Not<string>) === "hello" is always going to return false and we can emit this as a warning during our type checking procedure (technically a linting warning, based of type information).

In summary: when left subtyping `Not` then we test disjoint and when disjoint with `Not` we test subtyping.

Some other details

When Not is the super-type, it is never a subtype.
In many usages Not drops down to being a any type
The really important thing is through set theory to be assignable to Not<0> implies that it is disjoint from 0. It is not simply denying the 0 type but all types that have 0 as a member
Most types are of the form T & Not<U> where we have properties of T but we want to remove the properties of U.

Because this type reuses the intrinsic system and the fact that subtyping and disjoint are already implemented. This addition is a <100 LOC addition.

See the disjoint function here

Disjoint is currently not implemented for functions and objects. You can read issue if you want to tackle it!

Aside: `Not` vs `Exclude`

Exclude can be used to negate members. However it only works for unions and so only simple cases. The Not type is much more complex and can pick out constant from larger types.

From a language purity standpoint intrinsics are not great. Common utilities like this I think are fine though as long as the internal type list doesn't break 100 it should be fine.

You can see the negated types feature request in TSC here which would be similar to the Not type described above.

Numeric restrictions

With the new negated types, we have Not<0> and Not<NaN>, which can be helpful in restricting APIs to deal which can help when doing math without the crazy-ness of floating point mathematics and with the x !== 7, the Not<7> type can help catch disjoint errors based on single values.

However, there is still some common cases involving numbers where we could have more accurate types. For example we know that the result of a Math.sin call will never return a 2 as the range is between -1 to 1 (unfortunately JS does not support the complex datatype).

So similar to the Not addition we can add some more behaviour here, so without further ado we have ranged types.

Implementation

Similar to negated types we can add this with intrinsic. We implement this with two new intrinsic types LessThan and GreaterThan.

Unlike Not, these intrinsic have a bit more involved implementation that can't reuse much. When subtyping we check that the argument as a range fits inside the parameter type, for disjoint we check that they don't overlap as ranges. Unfortunately this is quite manual.

Narrowing numbers

We can extend the narrowing implementation to also handle cases from inequalities.

function func(param: number) {
	if (param < 10) {
		// param.
	}
}
function func(param: number) {
	if (param < 10) {
		// param.
	}
}

And in the block we have param: LessThan<10> !!!

More on narrowing in the future. Hiding a lot of the complexities of <= and building up number types that have the form LessThan<floor> & GreaterThan<ceiling>.

Annotations and ranges

Similar to Not we will add LessThan and GreaterThan to the root. From these primitives we can define ranges as intersections of these types and we can make endpoints inclusive with unions.

type GreaterThanOrEqual<T extends number> = T | GreaterThan<T>;
type ExclusiveRange<F extends number, C extends number> = GreaterThan<F> & LessThan<C>;
type GreaterThanOrEqual<T extends number> = T | GreaterThan<T>;
type ExclusiveRange<F extends number, C extends number> = GreaterThan<F> & LessThan<C>;

These are defined in the standard lib.d.ts

The implementation for them is pretty nasty. It is nicer to deal with them in terms of ranges (for example checking disjoint-ness) but it is an abstraction that breaks down.

And we can actually use this type in the standard library, for example Math.sin returns a InclusiveRange<-1, 1>.

What you can do with them

We can do some pretty simple things with them for annotations. For example requiring that positive numbers in places.

The cool things come from narrowing and operations. For example we can catch equality as they have an implementation for disjoint-ness.

We can add more logic to our inequalities testing whether a range is always above or below.

Also transitivity kind of works (this is with the bounds not being known numbers)

`MultipleOf`

Similar to LessThan and GreaterThan, MultipleOf is another new intrinsic number type that can be used to narrow down on the value of numbers.

We again expose this through a type annotation such as MultipleOf<5> and also through narrowing with x % 5 === 0. I won't go through the details of it all here.

Integers

Integers can be created by MultipleOf<1>. This is because these number intrinsic are based on the number type which at runtime is floating point number.

Integers with arrays / strings?

My intuition is that this can help with array/string indexing. For example we know that if a value is in 0 & LessThan<this.length> & Integer, we know it is not undefined. This can help for indexing an array (or a string) in for (let i = 0 loops (which do come up). There is some problems with holey arrays though and so I haven't done added anything around this.

(although holey arrays seem user driven with delete, I don't think they can come from external APIs)

There are a lot of things we could extract from floats. It could assit with WASM generation where there exists whole numbers. But investigating all of these takes time and is a bit out of scope for the moment.

More restrictions with the `Literal` type

Another intrinsic type which I added ages ago is Literal. This constrains the user to only be able to write known constants to this value.

This can help with some things. For example you can lock down fetch calls to only use fixed URLs, custom element registration etc.

Unfortunately, this doesn't add any safety... but it could be some using in adding rules into the language. Here JavaScript will always let you create a property under the key of the current weather in the UK.

const obj = {};
Reflect.set(obj, await fetch("https://uk-weather.api").then(resp => resp.text()))

// Property always exists
console.log(obj.cloudy)
// okay sometimes this
console.log(obj.sunny)
const obj = {};
Reflect.set(obj, await fetch("https://uk-weather.api").then(resp => resp.text()))

// Property always exists
console.log(obj.cloudy)
// okay sometimes this
console.log(obj.sunny)

but we can create a better way using Literal<...> types

class ReflectingOnReflect {
	static better_set(
    obj: Literal<object>, 
    property: Literal<string>, 
    value: any
  ) { 
    // ... 
  }
}
class ReflectingOnReflect {
	static better_set(
    obj: Literal<object>, 
    property: Literal<string>, 
    value: any
  ) { 
    // ... 
  }
}

Testing and feedback

Hopefully you found something interesting. All of this is open source, you can play with them in the playground and if you find any bugs with the new experimental items, post them on the issue tracker.

And if you have found anything difficult to represent as types, that is still not covered by these new types, let me know in the comments below!

Direction of the project

While this stuff is fun, there is probably a number of you asking whether you can begin using this is projects. The unfortunate answer is that the ezno-checker is not currently usable on projects because of the lack of many TSC features or partially implemented features.

The checker has been crawling along slowly, passing more and more specification tests and working on more complex parts. I now somewhat confident in the foundation and that many of TSC's features have been somewhat reimplemented. You will see in the next two blog posts that cover mapped types and narrowing respectively.

While these experiments like this are interesting, they don't necessarily work perfectly under all the edge cases. In some cases a number of the features have been relegated behind the --advanced-numbers flag.

The biggest experiment in the checker has been side-effects recording and replaying. There have been some huge progress with respect to that, most of which I have not currently shared (but maybe soon). A lot of that stuff is currently very delicate and so I think the next steps will be to put that behind a flag. Once that is mostly out of the way for default, the actually usability can be attempted.

Currently allocation of resources is a bit of a constraint, but that only affects the timelines of these things and not the achievability. However there is still more to share and I have a blog post on mapped types and one on narrowing (on more than just number types) that should be ready for publishing soon!

This blog post was sent out to sponsors early. As well as supporting the development of the project you also get early access on information on the project and access to the LSP! As well as monetary contributions, you can also contribute code. There are some good-first-issues and many have already been completed by some awesome contributors!

Sets, types and type checking

2024-10-30T00:00:00Z

In the process of building a type-checker I have learnt a lot of the details about the theory of types and sets. With all this information I thought I would unpack all the details I have encountered along the way.

This first part of this post explains why type-checking exists and what the abstraction of types enables. After some justification for type-theory, we delve deep into what types are and details on the common constructions. In the finale we will put the definitions to work and explain the implementation of type checking including operations on types.

While not many people are building type checkers, a background of the implementation can help with being more effective when using (or dealing) with a type checker.

I have a few upcoming posts on more advanced and niche type features. So I thought I would put together this post that lays the foundations so that I can refer to it the more advanced things. It has been quite fun recapping on some of the parts I have built along the way.

Some of this is relevant to TypeScript (and other structural systems) but there are parts that are apply generally to programming languages of all kinds of implementation of "typing". And in some of the wall of text there is real life analogies, interesting diagrams and some interactive gizmos!

This is my interpretation of type theory based on the things I have worked with or built. Similar to the type checker, it is not complete and this post may evolve. If you think I have not covered anything in this general post or alternatively have any cool resources on type theory, you can share them to the comment section at the bottom of this post!

Types provide information

Before delving into the details, I thought it would be good to justify why the theory exists for all those asking the point of type theory?

The simplest way to put it is that what: types provide information about the structure of expressions, what parts of the programs take as input and what parts produces as output.

Information for errors in invalid programs

The main association of "types" is in type-checking. Once the program types have been resolved, each individual type can be compared between other types and usage of the value to check that the program is sound.

For example here we find that a number is being multiplied by a string. In this example multiplication on strings is not defined and so we have a problem with the program. Information about this type can either be surfaced at runtime or like the below ahead-of-time. Either way, denoting the type of an expression has helped find an issue in the declaration of this program.

Information for build optimisations

I will not go too much into it here, but type information is useful for making decisions for the program to run with less overhead. For example in C, giving type information about numbers instructs more optimal registers to use in the CPU.

There are slightly more esoteric behaviours such as mutable aliases in Rust. But will not go too much into optimisations here and focus on checking aspects of types.

Information for the programmer

Humans like characterising things and types provide a way to do that within programs and code. Not everything can benefit from the additional syntax, but for data based problems, writing out the types before the logic can help better structure the program and understand what needs to be accomplished. When coming back to code we can look for the key data structures and from there see how the methods connect up. Things like adding return types to function can help understand what the function generally does without having to delve into the full definition.

Types also help when understanding the unfamiliar. Whether that be a standard library or an external library, providing type information can help to see what some. For example the Rust std.rs (and docs.rs for a individual libraries) is a encyclopaedia of structs, enums and traits which all come under the umbrella of type information.

Additionally types help with making changes to code. If we change a function, then we can see how that change affects code that it was relying on it. Additionally while changelogs are helpful, this abstraction is much more mechanised for upgrading 3rd party libraries.

I won't go fully into the ergonomics of types, but I think it is good to state that types are not just for the machine.

Types are not the only components for the abstraction program, other properties such as intermediate representation (IR), control flow graphs, concrete and abstract-syntax-trees etc parts all with for different things for parts of program synthesis (program synthesis = a machine understanding what a program does)

What is a type?

Types describe data. Data represents things, properties, maps from data to other data, information about the relation of things. Types allow us to categorise data and this discernment and grouping allows reasoning.

This categorisation is done based on properties. Here I use the word property in the abstract sense, something measurable of the data, some way about the data works in regards to the rest of the system. (A sort of extension to fields and data members as in programming languages).

Some real world properties could be:

Is red
Puts out fires
Is a fire-truck

Each of these properties is tailored to the language. For example some languages have "symbols" which are basically unique values for a program. We could define a type that requires the exact symbol. Similarly other languages have the concept of asynchronous functions, this property could be encoded into the features of a type. In Rust we could have a function which requires the inputs to be of type &'static str which are string slices valid throughout the lifetime of a program. We could also have restrictions such as the type implementing Display meaning that it has a can be printed to a user.

This encoding of properties as a types provides the information to reason about what data is throughout the lifetime of a program and so in the the cases show above can help catch issues and optimise programs ahead of time.

Types are slightly different to sets in which they are based on construction rather than predicates. Construction refers to building something given some input. We have some data we can form a member of the type via some constructor. I will write about the differences in a future post. But for the most part sets can be considered equivalent to types, with the same rules and uses.

Members

In the world there exist entities that fall under these properties

Tomatoes, the YouTube logo and 🌉 are all red
Carbon dioxide and 🧯 can put out fires
🚒 is a fire truck

To say that a value satisfies the property is often know as a typing judgement and we often write it as colon. For example ⊢ 🚒: fire truck, or in the sense of programming language ⊢ 5 : number.

Note you will see the : come up a lot in type theory

Facts about members

Like sets, types can be by description have an infinite number of distinct members. However when it comes to programs on deterministic Turing machines (computers) there are only a finite amount of numbers that can be represented due to limitations of memory. Benefiting from exponential combinations, while finite it turns out even with 8 bytes of memory you can still cover a very large amount of numbers.

When encoding these sort of types we could treat them as collections of the individual members. However, storing the members of u64 in some list would result in 147 000 000 terabytes of data and individually I can't begin to count the CPU years it would take to iterate through all members of the data and comparing that each entry is valid for the program. So instead for types like this it is easier to represent u64 as a number type that is unsigned (no negative values, zeroed out bytes is the number zero) and is 64 bits wide. With this information it covers all members and we can more efficiently reason about the value, rather than querying each entry.

With members, we have a implication from a valid judgement. If we have that x = y and that x: number then we can conclude that y is a number. This is generally a property you want to uphold when defining equality. The equality operator should be testing the all properties are equal and thus we can say they have equal types.

Additionally a random last fact for members: It is harder to find members for types which have more properties (and so in general there are less members for types with more properties).

Fundamental types: `any` and `never`

Two of the most fundamental types are any and never. These are polar opposites of each other.

any can be considered as a type which doesn't have a single required property. We can say a fire truck, a spoon or the string "Hello World" is an element of any thing because there are no properties to satisfy. (If it helps this is a case of the vacuous truth with respect to the number of properties to satisfies being vacuous). Therefore any contains every value you can construct.

any exists in TypeScript but has slightly different behaviour in some cases (explained later). But it is not just a property of the dynamically typed languages, Rust also has an any type which works slightly differently but is used for general data. There also exists unknown which is any without the strange subtype behavior.

never on the other hand has all the possible constructed properties. To be an item of never is to be in London and on the moon and a single colour and be blue and and green and purple. Because it requires every property there will exist two properties that are mutually exclusive. (this will be covered later). You cannot be a single colour and blue and green at the same time. Thus there is no constructable item that satisfies never*. Thus we can say that the collection of items that satisfies never is empty.

never can be used to represent places where functions never exit. For example in an infinite loop or something that throws an exception. It exists in TypeScript under never and sort of in Rust.

As will be explained later these two types have very important positions in the hierarchy of types

I don't think never is a good name. Maybe impossible or CannotExist is alternative better names for thinking about this type. We can reason that it has no members as if ask for something impossible, then you are stuck. You can't pass 5, or a "Hello World" or a Boat as they all exist.

Conjugations of types

There are two forms of binary conjugations of types and their properties. Binary here meaning that there are two types being joined. This results in its own type as so we can form a recursive definition of a type.

For implementing the definition of this type: you can represent both of these as a vector or a binary tree of other types.

And types (set intersections)

An and / & type represents a intersection of two types. This is equivalent to saying that members of this type satisfy properties of both the LHS and RHS. For example we can define a rainforest as having a continuous tree canopy and high humidity. For an item to be considered to a rainforest it must both satisfies the properties of having a continuous tree canopy AND having high humidity. If either of this conditions is unmet then it cannot be considered under this type.

type RainForest = ContinousTreeCanopy & HighHumidity.
type RainForest = ContinousTreeCanopy & HighHumidity.

A definition of this type will be unique, however and types are symmetric. A & B is equivalent type to B & A.

JavaScript objects are why we have intersection types

You may be thinking what are the use cases for an intersection type?

The first thing to know if that in TypeScript, object type annotations only specify their required properties and can allow members to have more than specified. For example ⊢ { a: "hi", b: 4 }: { a: string }. Just their definition doesn't require there to be an exact amount of properties.

Many languages don't have intersection type. Often the intersection types in set land are represented as products/tuples (fixed sized lists) in other languages.

For example [{ a: "hi" }, { b: 4 }] has the equivalent amount of data as { a: "hi", b: 4 } (it even retains the order, which is a strange property of JS objects), but we want our type system to not allow .push on this item. Hence we have intersections and don't resort to tuples to maintain the same semantics as JavaScript.

With the intersection type we can build up objects { a: string } & { b: number } = { a: string, b: number }. While

#insert the colour SVG thing here

Also most of the time, they are stored in flat representation. For example while we can represent a structure definition as an intersection of single property objects, grouping them together just makes the implementation easier, especially in other languages that have to consider type layout which requires knowledge of the adjacent properties is required. So instead and types are used to extend existing types. If we have existing Type we can extend it with more properties with the and type operator type Extended = Type & { property: string }.

In some languages this can be considered as a refinement type. We have a base type but we adjoin extra required properties on-top of the LHS.

Reducing intersection types

Intersection types have a diagonal identity. A & A = A, so if we try to build an intersection with the same type we can just return the initial result. Additionally never & A is always never. While there might be members for A, we don't have any members for never and so the nothing exists in never & A and therefore it is equivalent to never.

Additionally there are some other reductions we can do (these include operations that will be covered later). The first is that if A is a subtype of B we can consider the smaller type (subtype): A & B = A. For example 4 & number = 4. Additionally if type A is disjoint from B we can consider the result to be never: A & B = never. For example number & string = never.

Why the intersection with `any` is always `any`

An important point here is when any & X is considered. From a set perspective it should say that any is always the super type (larger than X) so any & X is always X . However TSC has slightly different meaning for any in places and actually considers it a subtype as well (covered later). Therefore for TypeScript any & X is actually any. I am not sure if the subtyping reuse is why this is the result, or whether it is specified specially here but the general take that in cases of any, the result is mostly any.

Or types (set unions or sum types)

An or / | type represents a union of two types. This means that this value can either have the properties of the left type OR the right type. For example we can define a train as either a CargoTrain or a PassengerTrain, which we can represent as type Trait = CargoTrain | PassengerTrain.

Note that this is not exclusive or relation. We can define sum type such as blue | wet then the sea blue so it is considered an entry, it isn't necessarily to check the wet property. Normally most sum types are between disjoint properties so this is not an issue.

Similar to intersection types, this can be recursive relation A | B | C | D. We can consider this as a list of a binary tree ((A | B) | C) | D. This can be useful for things as type TrafficColor = "red" | "amber" | "green" (at least in the UK).

Equivalence rules

Similar to and types there are many equivalent representations:

They are symmetric A | B = B | A.
Satisfy a diagonal identity A | A = A.

Additionally, never types in unions can be removed/collapsed. Aka Is it going to rain today or does it never rain in England is equivalent to Is it going to rain today (and so is collapsed down to the left, because the case on the right is never held). Therefore constructing types we can just keep the never cases A | never = A.

You can think of any as being a union of all possible types. Using the symmetric to rearrange and the diagonal identity we can therefore see A | any = any

Additionally like intersection types, we can reduce it based off of disjoint and subtyping. If two members are not disjoint we can pick the larger type. For example string | "hi" can be considered as the larger type string. (The interesting point is that intersections we can pick the smaller type (subtype) whereas with union types we pick the larger type (super type))

Ezno does not do eager reduction or test subtypes when. I have found that at some point causes more problems than it solves just for minor visual clean up and potential to do slightly less subtypes per usage. Especially when you consider the combinatoric scale of checks when comparing all the items large unions.

Tagged vs untagged sum types

Rust has both untagged unions with union and tagged with enum. This tag is called the discriminant and retrieved with the titular discriminant function (which extracts tag data under-the-hood). This value / tag is unique between each defined variant.

In JavaScript each variant (before optimisations) has a tag. You can consider it globally tagged. This isn't exposed through memory but operations like typeof and instanceof are based reading this tag (some of the time). In this crude example we can consider the first four bits of a value to include its tag. For example booleans are 0, numbers are 1, strings are 2 and objects are something different all together maybe referencing something else in memory that contains there shape.

false    = 0b0000_0000
true     = 0b0000_0001
5        = 0b0001_0101
"hi"     = 0b0010_1010...
{ a: 1 } = 0b1010_0000...
false    = 0b0000_0000
true     = 0b0000_0001
5        = 0b0001_0101
"hi"     = 0b0010_1010...
{ a: 1 } = 0b1010_0000...

If you are interested in learning more about this I would recommend posts on V8 about how it represents objects.

Nullability and error handling

One of the more common usages of sum types is for representing some data which can be missing. For example we can represent the colour of a ships sail as Color | null as some boats do not have a sail.

declare function getSailColor(boat: Boat): Color | null
declare function getSailColor(boat: Boat): Color | null

In Rust we have Option<T>, which is equivalent to T | null

We can expand on the null variant with an type which has more information about the why the first value was not returned. For example we can expand on the above function with more information

declare function getSailColor(boat: Boat): 
	Color | { message: "Boat has no sail", boat_kind: BoatKind }
declare function getSailColor(boat: Boat): 
	Color | { message: "Boat has no sail", boat_kind: BoatKind }

In Rust we have Result<T, E>, which is equivalent to T | E

(I think) In Golang the is no definition of a sum of types. Instead you can use the fact that every value is adjoined with a null member. Instead to represent something like the error type above can use pairs of these values to represent sum types instead.

Combinations of conjugations

Note that both of these are subject follow the roots of boolean algebra. Each operations is are symmetric and importantly they distribute.

(A | B) & C = (A & C) | (B & C)

In word terms: an apple or banana that is large is the same as a banana that is large or an apple that is large.

All types are unions of intersections

You can think of as all types as being a union of some intersection. These are cases of necessary conditions. The { a: string, b: number } | null can be thought of as two cases *object* or null. In the first case there are two necessary conditions that obj.a: string and obj.b: number. In a sense the reduction above is forming the minimum cases.

Parametrised types

Often called generics and used to model structures that are dependent on some type: parameter can be added to various definitions to specifying insertion of types at certain places. This can be useful for providing templates for types.

These are sometimes considered PI types or "big and types", which I might cover more in a future blog post.

Array is an example of a type that has some relation to an inner type or works over many types. While we could consider the items of an array to just be any we can refine this type by detailing the types of individual members.

One way would be to define a type for each kind of an array.

interface StringArray {
	push(a: string);
	// ...
}

interface NumberArray {
	push(a: number);
	// ...
}

// ...
interface StringArray {
	push(a: string);
	// ...
}

interface NumberArray {
	push(a: number);
	// ...
}

// ...

This has the problem that we now to duplicate the definition and each time are tweaking the same fields. It also means that if we create a new type such as Boat, the user has to create a whole new specific definition for Array.

Therefore we use generics to denote a sort of template for our general type

interface Array<T> {
	push(a: T)
}
interface Array<T> {
	push(a: T)
}

We will see later how, but we can create equivalent definitions above by instantiating the above with type StringArray = Array<string>; type NumberArray = Array<number>;. We can also solve the problem when introducing new types as we can get our array of boats with Array<boat>.

We can add parameters via chevron notation <T>. They can be directly added structures like interface in TypeScript and on struct and enum in Rust. We can also add them for any type using type aliases. Our Option type from Rust we can represent as type Option<T> = T | null.

These are often used for general types, things that a bit meta that represent modifications of things in nature rather than existing in nature themselves. For example Array<T> for a list of values with an unspecified length, or Set<T> for the same without order specified and with the condition that values are uniquely added, or Map<K, V> for associations for data. (the latter two can be defined in terms of Array<T>)

Often use T as the first letter in type. From there additional go up the alphabet into U etc. For things like Map we use K and V for Key and Value respectively. Sometimes I for item. But we normally have the convention of being a single character and like most type names for the first character to be uppercase.

Parameter types

The parameters that we can give to our types are types themselves. They have names we can reference in annotations, used the intersection and union types shown above and also in everything following this.

Generally these types extend any. We can change that though with the T extends Type syntax in TypeScript and with T: SomeTrait in Rust. For example for our Set<T> definition we can require that the parameter T implements equality and maybe a hash operation (heuristic for equality) for the set to be to valid.

Partially applied generics

With regards to parametric types we can instantiate them with type arguments. For example we can have an array but expand to that the elements of the array are strings with the Array<string> syntax.

This string tells us that if we have a expression of this array type then getting an element of it will return a string. It tells us more though, for example push must take strings as inputs for our array to still all be strings. it tells us that the functions passed to .map, .every, .some must all be defined for strings so that they can be evaluated against the elements.

This point is important: a type parameter may be used many places including nested in other types such as parameters for functions on methods. Therefore we need to handle these generics carefully as not create lots of instantiations based one argument.

Lazy lookup of generic types

In the above example we looked at the Array type, I have revealed a few more fields on the type to show how type parameters can be appear multiple times in difference places.

interface Array<T> {
	[item: number]: T;
	push(t: T);
	map<U>(mapped: (t: T) => U);
	// ...
}
interface Array<T> {
	[item: number]: T;
	push(t: T);
	map<U>(mapped: (t: T) => U);
	// ...
}

When we have a reference such as Array<string> we need to produce an internal representation of this specific type of Array. One method would be to take the Array definition and walking the structure, copying out each member substituting any references for T with string. Such would result in the following.

Array<string> = {
	[item: string]: number;
	push(t: string);
	map<U>(mapped: (t: string) => U);
	// ...
}
Array<string> = {
	[item: string]: number;
	push(t: string);
	map<U>(mapped: (t: string) => U);
	// ...
}

However, we can speed up this step and instead form a PartiallyAppliedGenerics item. PartiallyAppliedGenerics is a pair of a type (with generic parameters such as our original unspecified Array) and a Map with arguments (Vec<(TypeId, TypeId)>). So Array<string> becomes PartiallyAppliedGenerics { on: Array, arguments: { T = string } }. This form is a lot more efficient as we just need to create a single wrapper Type and don't have to inspect or copy any of the internal structure when we instantiate a specific version of an Array.

(aside from checking that it has matching parametrised and type arguments match extends clauses).

I think partially applied is the correct term, we fix some types but don't fully evaluate everything.

This however means that using this type is a little bit more difficult to use. We can't just access a field in the regular field as it hasn't be specialised with the arguments.

So now when accessing information (a property, calling a function etc) we lazily compute it by unwrapping the pairing. Arguments is collected in a table like structure called GenericChain and the left side is run unfolded. If a generic is found, the table is queried (in the checker I am building it is with the method .get_argument_covaraint) and only the closest item needs to be substituted.

PartiallyAppliedGenerics wraps things like generic interfaces, methods which inherit parent terms and anonymous objects. Note that type aliases is not in this list as in TypeScript for type aliases (type AliasName<T> = ...) instantiating them eagerly substitutes them into the RHS, rather than leaving them as AliasName<Argument>.

All type arguments on structures are considered restrictions for usage, so in type terms "covariant".

This mechanism is reused for another feature around closed over variables. But that is for another post.

Special root types

Products of types / tuples / pairs

We can form concatenations of types with products. For example we can accept a number and string for a function with a tuple. This is normally defined by a comma separated list within either parenthesis () in Rust or [] brackets in TypeScript.

function func(a: [number, string]) {}
function func(a: [number, string]) {}

Products are different to type intersections as explained above for JavaScript objects. They do have equivalent properties though, but that is a for a future post.

Tuples can be heterogenous (of different types) whereas arrays normally are homogeneous with the same type (up to sum types). In TypeScript Array can heterogenous and so the tuples are not so much a completely different type but instead a special kind of array with more properties for the exact length and exact types at exact indices.

In set theory, this is known as Cartesian product.

Objects/structs are a special product types

With tuples we normally access items by their zero-based index. In the above example if we want the number data passed we retrieve it with a[0] (or in Rust a.0).

Structs and objects extend this idea by referring to these indexes by string keys (instead of numbers). In Rust

struct X {
	first: number,
	second: string
}
struct X {
	first: number,
	second: string
}

is equivalent to the (number, string) type but it has the additional compile time metadata which allows us to refer to the fields with more natural names (you can see how this especially helps when we have fields of the same type).

In TypeScript, it inherits that objects information exists at runtime and so the objects are slightly more complex. They also allow addition of properties that were not present at the time of declaration so the types become slightly more complex.

Functions

Functions are a fundamental type in the theory. They represent transformations between lists of values to other values.

Function types abstract the code definition of the expression function and primarily focus on what the inputs and outputs are.

A function is defined as a sequence of parameters, which have required types. The types for inputs is called its codomain and the result of applying/calling the function is the domain.

Parameters are the variables. Arguments are the values passed to the function at the call-site Ahem

Generic parameters

Similar to generic types, functions can also be parameterised with generics. Sometimes these values can come from above (such as the T in Array.prototype.map). Some are local and are set at the call-site. In later parts we will see how these come to take a value and how they affect the result. But for now these are considered on the same level as parameters with a fixed type.

There are however some special kind of parameters which make things a little bit trickier.

Default arguments

Default arguments are values used in place of an argument missing

function add(a: number, b: number = 1) {
	return a + b
}

add(1, 2) // 1 + 2 = 3
add(1) // 1 + 1 = 2
function add(a: number, b: number = 1) {
	return a + b
}

add(1, 2) // 1 + 2 = 3
add(1) // 1 + 1 = 2

This is equivalent to after some de-sugaring

function add(a: number, b_before_: number | undefined) {
	const b = b_before ?? 1;
	return a + b
}

add(1, 2) // 1 + 2 = 3
add(1, null) // 1 + 1 = 2
function add(a: number, b_before_: number | undefined) {
	const b = b_before ?? 1;
	return a + b
}

add(1, 2) // 1 + 2 = 3
add(1, null) // 1 + 1 = 2

For reasons it is simpler to write the internals in terms of default arguments inline with language and to not apply this transformation. But you can see from this rewriting how it functionally works.

Note that the parameter changes type here. Despite the annotation, the resulting function parameter type is union-ed with undefined which is the value it has in JavaScript when an argument is not passed

Variadic arguments

Rest/spread/variadic parameters is the fact that at a call site it can take a non-fixed amount of arguments such that the later are collected in a collection type like an array

function getLastArgument(...a: Array<string>) {
	return a.at(-1)
}

getLastArgument(1, 2) // 2
getLastArgument(1) // 1
function getLastArgument(...a: Array<string>) {
	return a.at(-1)
}

getLastArgument(1, 2) // 2
getLastArgument(1) // 1

This is equivalent to after some de-sugaring

function getLastArgument(a: Array<string>) {
	return a.at(-1)
}

getLastArgument([1, 2]) // 2
getLastArgument([1]) // 1
function getLastArgument(a: Array<string>) {
	return a.at(-1)
}

getLastArgument([1, 2]) // 2
getLastArgument([1]) // 1

Similar to default arguments and especially because of the call site transform for a list this again does not apply this transformation and instead this property is encoded into the function type.

Multiple parameters and currying

We can consider it under the currying to transform

Repeating myself again. It is closer to the language to treat the parameters as a list. While finding more canonical representations can help simplify cases. Here we want to preserve a form closer to the syntactical representation as it helps in printing and calling. In a language such as Lean the forms are equivalent (the latter is desugared into the former).

(a: string) => (b: string) => a + b
(a: string, b: string) => a + b
(a: string) => (b: string) => a + b
(a: string, b: string) => a + b

Currying is named after the great Haskell Curry a renowned logician. Curry is also great too.

Free variables and closures

Sometimes a function can use reference something outside of its inner scope. When a variable is used above it is considered "free" as it is not bound by a parameter.

Difficult situations can arise when these can be mutated

let b = 2;
function add(a: number) {
	return a + b;
}

add(1) // 3
b = 6;
add(1) // 7
let b = 2;
function add(a: number) {
	return a + b;
}

add(1) // 3
b = 6;
add(1) // 7

Closures are the level above free variables. It is where a function is returned that uses a free variable in the above scope.

function closure(a: number) {
	return function () {
		return a++
	}
}
function closure(a: number) {
	return function () {
		return a++
	}
}

Closures are hard as they implicitly store state. Unlike objects we don't construct a definition for what this data is like. It is sort handled implicitly build from the context. Closures have problems in lifetime based languages and so that is why the move keyword exists in Rust.

I think free-variables is the correct terminology

I have a further work on closures but will skip over for now.

Other functional properties

Async functions can be considered to return their result as a Promise (in TypeScript) or a Future in Rust. These are considered as just modifications to their return type without much else effect.

Okay enough of the complications added by functions, back to type theory

Conditional types

We can define types (such as this return type for a function) whose properties depend on a condition

function func<T>(t: T): T extends string ? "is string" : "not string" { .. }
function func<T>(t: T): T extends string ? "is string" : "not string" { .. }

Here the type is T extends string ? "is string" : "not string" with the condition T extends string (which is a typing judgement slightly abused as a predicate here).

As we will see later, when we call this function we substitute the return type. In this case we evaluate the judgement extends string against our resolved value for T. For the condition we can resolve one of the branches or both if this proposition is not resolved discretely into either true or false.

This type is a union type with a little bit more information of what the result is. If we drop the conditional information it can be treated as "is string" | "not string" in many cases.

In fact sometimes A | B can be thought of as *free condition* ? A : B, where the condition is unknown and unresolvable inside the system.

Sometimes, when using this type we might want to carry the condition data forward. If we index the first character here (with item[0]) we can result with the more informed T extends string ? "i" : "n" (rather than the weaker "i" | "n"). This can sometimes better narrowing.

Implied conditions with example in Rust

Rust does not have conditional types. You can imitate this with generic associate types but not in the same way as you have to have implementations for each possible input type. It is not possible to cover all the types like in TypeScript because of colliding implementations with impl. Even with #![feature(negative_bounds)] it does not seem possible.

Conditional types distribute across their condition

An important thing is that for conditional types, if there condition is a sum type, then it is distributed.

type MyConditional = (A | B) extends C ? X : Y;
// Immediately interpreted as
type MyConditional = (A extends C ? X : Y) | (B extends C ? X : Y);
type MyConditional = (A | B) extends C ? X : Y;
// Immediately interpreted as
type MyConditional = (A extends C ? X : Y) | (B extends C ? X : Y);

This is an immediate transformation and not an equivalent type. This is also applied during generic substitution

This distribution added (I think) enables some fancy filtering of types. In the future I will show another way that this is possible.

You can use the undistributed version by wrapping the two sides in unit tuples. [A | B] extends [Type] ? X : Y;

Type hierarchy

1 ∎ any contains everything below it. 2 ∎ here is where type classes (called traits in Rust which I think is a good name) exist. They are big unions of nominal types. 3 ∎ is where smaller unions exist which are useful for representing data up of 4 ∎ nominal types. Boolean takes two to build its union. But number (which is normally a 64-bit floating point number has members in the union). Exact object types could be considered here. 5 ∎ is useful for tags and small amounts of types. 6 ∎ is dependent types. we know all the information about them and there is one entry. 7 ∎ the never type. Can be considered as an intersection as anything on the row above

I haven't included functions, generics or some of the other previously seen types but just the basics to see how this lattice sort of representation holds

Operations on types

Subtyping

Subtyping around set types is basically asking this question:

Does this set fit into this other set?

When and why subtyping

Before asking how to do subtyping we need to ask when it is done and why it is required. The following is a function with a parameter with type number.

function addOne(a: number) {
  return a + 1;
}
function addOne(a: number) {
  return a + 1;
}

A call to this function is only valid when it is valid to substitute/swap the arguments for parameters.

function func(parameter: string) {
//   		  ^^^^^^^^^
//            parameter
}

func("argument")
//    ^^^^^^^^
//    argument
function func(parameter: string) {
//   		  ^^^^^^^^^
//            parameter
}

func("argument")
//    ^^^^^^^^
//    argument

When we call this function we ask whether "argument" is in the string set (technically the singleton type is a subtype). The rules for which will be shown later, but in this example it is simple to see this call is valid as the string "argument" has all the properties of being a string.

From this we know that this call is valid. However if it were the case that we had func(4) we find that type 4 does not have the properties of being a string and so here we would raise a type error.

There are many times questions this "subtype" question is asked, so here is a long list of all the places where a subtyping happens. In each case I have linked to where is_subtype (or a wrapper function) is used in the ezno-checker.

The first is in return types. When we return a value, we check that it is a subtype of the annotation specified in the function declarations.

function func(): number {
	return "hi"; // "hi" < number
}
function func(): number {
	return "hi"; // "hi" < number
}

Variable initial values (including default values) are another case

const x: number = "hi"; // "hi" < number
const x: number = "hi"; // "hi" < number

Assignments after declaration (including properties) is an extension of the case above

let x: number = 4;
x = "hi"; // "hi" < number

let y: { a: number } = { a: 4 };
y.a = "hi"; // "hi" < number
let x: number = 4;
x = "hi"; // "hi" < number

let y: { a: number } = { a: 4 };
y.a = "hi"; // "hi" < number

Others include: satisfies, Try-catch (special to ezno-checker), Function overloads,Generic arguments to parameter extends in type annotations, This constraint (special parameter case), Type based keys (which will get more of a mention in a future post), Operator checking (this uses a wrapper method) and of course the above mentioned Function arguments to parameters.

Subtyping allows us to reason whether about a substitution or assignment is valid. Like the fact we do not test each individual entry, subtyping of function parameters allows skipping all the individual checks already done in the function body.

function func(x: string) {
	const a: string = x; // string <= string here
	const b: string = x; // string <= string and again
	const c: string = x; // string <= string and here
}

func("a") // Single check against parameter (aka does not result in 4 checks)
func("a") // Single check again
function func(x: string) {
	const a: string = x; // string <= string here
	const b: string = x; // string <= string and again
	const c: string = x; // string <= string and here
}

func("a") // Single check against parameter (aka does not result in 4 checks)
func("a") // Single check again

How to do subtyping

Subtyping is the asking of a question whether one type is contained in another two. We can extend our Venn diagrams with an extra dimension and consider by considering one type being overlaid by another type. If I have it all working correctly, you should be able to drag the sliders to move the sets each other. For the overlaid set to be a subtype we want to see no red colour here at the end of the end of the translation. We see that sets on the right end with items that don't fit into the base type and therefore is not a subtype. If we have no red, then we have that the top type is a subtype.

Note that this is a relation not an equality. At the end of the transition we can have left over yellow.

Sometimes the notation <=: is used to denote this relation (the underline to denote that they can be equal and the : which crops up all around type theory).

X is a subtype of Y is equivalent to saying X is assignable to Y.

From the above diagram we can conclude from the top-left case that 4 is assignable to number so 4 <=: number but not the top-right case "hello" <=: number. Similarly on the bottom row, from the left side we find that number <=: number | string but not the other way round in the bottom right case number | string <=: number.

Subtyping is recursive

Because of the structural system, an individual test for relation can end up in many nested subtyping questions.

Sometimes cases can occur where subtyping can end up being cyclic. For this case a list of "currently checking pairs" is built so that subtyping can exit and not stack overflow.

The details

Here is the table specification

Left	Right	Result	Also	Why
Type X	Type X	Subtype		They are equivalent
Primitive	Constant of primitive	Subtype
Constant of primitive	same constant	Subtype		Equal (same as first case)
Constant of primitive	anything else	Not subtype		Only that value
A & B	X	A subtype X && B subtype X => subtype
A \| B	X	A subtype X \|\| B subtype X => subtype
A	X & Y	A subtype X \|\| A subtype Y => subtype
A	X \| Y	A subtype X && A subtype Y => subtype
Parameter with type constraint	*	Subtype	Set parameter
Parameter	Parameter	Subtype	Set parameter
`never`	*	Not subtype		Nothing is in the empty set
`any`	*	Subtype
*	`never`	Subtype
*	`any` in TypeScript	Not subtype
*	`any`*	Not subtype
functions	functions	explained below
objects	objects	explained below

From the first row in this table we see that subtyping is a reflexive relation. This sort of thing is the fast path of subtyping where we can skip any inspection of types.

Note when we have a conjugation of the left hand side of this relation its logic follows its name. However when they are on the right hand side it gets swapped.

Function subtyping

Sometimes functions end up on either side of this comparison. For example call-backs in parameters

function map(array: Array<number>, cb: number => string): Array<string> {
	// ...
}

map(() => {})
//  ^^^^^^^^ subtyping here
function map(array: Array<number>, cb: number => string): Array<string> {
	// ...
}

map(() => {})
//  ^^^^^^^^ subtyping here

Similar to standard values, we also need to check that the function is also valid to be passed here.

number => string <=: *our function*

The first property to check is that the function on the right hand side has less than or equal to the same number of parameters. If our function has more (required) parameters than the left-hand-side it means that we could call out function with missing data. If we have less then it will be fine as despite calls with extra arguments, it doesn't produce a runtime error, the arguments are simply discarded.

In Rust which is much stricter the counts have to be equal.

One of the most interesting things is that: Comparing parameters happens in reverse. If we have the following.

((a: "hi"): number => 2) satisfies (a: string) => number
((a: "hi"): number => 2) satisfies (a: string) => number

If we compare the types of the first parameters in the normal way for the rest of the types like "hi" <=: string , then we see it is valid. However this function on the left-hand-side is not (a: string) => number as it does not accept the whole set of strings as inputs (or as its codomain).

For example for the following, if we subtype the same direction we end up with an error when we call it with argument 5

const func: (item: string | number) => number = (item: string): number => string.length;

func(5)
const func: (item: string | number) => number = (item: string): number => string.length;

func(5)

The fix for this is to swap around the conventional subtyping operands. So instead we subtype string <=: string | number which holds in the revised below case.

const func: (item: string) => number = (item: string  | number): number => typeof item === "string" ? item.length : item;

func("Test")
const func: (item: string) => number = (item: string  | number): number => typeof item === "string" ? item.length : item;

func("Test")

This form is known as covariance and contravariance (I think)

Note that for default parameters and rest parameters the similar process is held.

The return type subtyping happens the normal way around. In the cases above we have number <=: number which is fine because of the reflexive property of subtyping.

Object subtyping

When subtyping an against an object, we iterate over all the keys in the left hand side and check that

It exists in the right hand side object
That it indexed result has the subtype

The following are all valid

{ a: "hi" } <=: { a: string }
{ a: "hi" } <=: { a: string | number }
{ a: "hi", b: false } <=: { a: string | number }
{ a: "hi" } <=: { a: string }
{ a: "hi" } <=: { a: string | number }
{ a: "hi", b: false } <=: { a: string | number }

This can mean that our right hand side can contain properties not in the right hand side. (explained near the start). This does cause some problems but that is for future posts as I think there is some progress that can be made in some of these cases.

However these are invalid

{ a: "hi" } <=: { a: boolean } // "hi" is not a subtype of boolean
{ a: "hi" } <=: { a: string, b: string } // missing a required property
{ a: "hi" } <=: { a: boolean } // "hi" is not a subtype of boolean
{ a: "hi" } <=: { a: string, b: string } // missing a required property

Collecting mismatches during subtyping

In the above cases for object you can see there is some reasoning why the the results are not subtypes. Two good features to aim for in subtyping are

No early returns: collect as many property mismatches as possible (this is further justified/explained at the end of this post)
Non-binary output: it is useful to know what exact property is missing. If the subtyping just returned a bool then it would miss out on information that could be helpful to the user in fixing this issue.

In the future there should be better printing of this result in ezno-checker. Currently some of the details are there but it needs some finishing touches

There is one other thing collected during this process, which you can read about in the inference section.

Subtyping: Nominal types vs structural type systems

One of the key differences between Rust and TypeScript is the fact that in Rust types are nominal. With nominal types while every type has properties, the difference is that the above subtyping cases is condensed to just is_subtype = A == B and does not look at individual properties in the definition of A and B.

This is done because in Rust types have layouts and so when substituting an object: the properties and discriminant values and offsets all have to line up. This can happen occasionally for types equal by definition but most of the time is just simpler to define it by instance rather than unfolding the definition.

And Rust can do this because they started the language and could add syntax for definitions of types and didn't add general object types.

In TypeScript being a superset of JavaScript we have to deal with users being able to create objects without a schema, thus structural systems are needed to handle for { a: "hi" } or JSON.parse being used as a value.

Apart from the primitives (string, number, boolean) there are no nominal types in TypeScript. If not for the equality shortcut, subtyping has to compare properties of sides to decide whether a value is assignable.

While nominal systems better define boundaries of values and result in less edge cases, structural systems can be simpler to work with especially in the design of something. Many times in Rust I end up having to define sum types for small amount of types which be cumbersome and could be made simpler with a structural system which unfolds the unions during assignment.

However, in Flow and Ezno classes are nominal

Disjoint-ness

Alongside subtyping, another less-referenced but equally important type comparison function is disjoint-ness.

While subtyping tests subsets. Disjoint-ness is a test that the intersection is empty. This is equivalent to saying there does not exist an element that satisfies both of the types or that the properties of the types are mutually exclusive.

If you consider these two in the Venn diagrammatic way you can consider two sets disjoint when there is no overlap.

Implementation

Similarly as subtyping we can resolve the results based on their definition

left	right	result	example
nominal type A	nominal type B	A != B	`"hi" === 4`
A \| B	X	A is disjoint with X && B is disjoint with X	`2 \| 4` is disjoint with `3`
A & B	X	A is disjoint with X \|\| B is disjoint with X	`number & 3` is disjoint with `2`
object	*	properties that reside in both having disjoint values	`{ a: string }` is disjoint with `{ a: number}`
never	*	true	...
any	*	false	...

Here nominal is the same as mentioned above can be a number, string or a constant 1, "hi", false etc. again for the type intersection and union the result ends up involving recursion.

Similar to the right hand side of subtyping, the or and and type use the opposite operator (this is probably because it is being framed from the point of view of disjoint-ness, if I reversed the name with "has some intersection" and changed the other cases then because of De Morgan identities it would use the result would be based on the same terminology as the inputs).

Unlike subtyping, disjoint is symmetric (disjoint A B = disjoint B A), which is why I only put in members for the first column because the results still stand swapping the left and right columns. This is why some of the code looks repetitive but it is the simplest way to account for both sides.

Functions and objects do not have disjoint-ness implementations are not implemented in the ezno-checker, you can help add them.

Uses for the disjoint relation test

This relation has slightly more esoteric uses that subtyping. But it can help resolve issues

Equality well defined

Most programming languages have an equality operator. In most systems you can only use equality is only defined for the same type.

In JavaScript we have an open book of values that equality operates on as equality always yields boolean no matter what operands.

We can however deduce some cases where say an object is compared to a number, { name: "Josh" } === 4 or even "s" === false (although this is handled slightly differently in ezno-checker for constants).

These cases are decided by the disjoint test. If they are not disjoint (aka have some overlap) then there must be at least one element that is in both and so this equality can be true or false.

On the other hand if we find that they are disjoint we know that there cannot exist a pair that satisfies the equality and so we know that this expression is always false.

This behaviour extends to not equal. If we consider the operation x !== y as !(x === y) we perform the same test (just the diagnostic prints that it is always true rather than always false).

Intersection types well defined

If you can remember all the way back in the beginning when we discussed some of the rules we around intersection types, we said if the two were disjoint then it was equivalent.

When writing a intersection type reference, it is probably the writers intention to write a valid combination. So in the following case it is a probably a good idea to warn them their construction is invalid. (if they did want the never type the could just refence it directly)

type MySpecialType = string & number;
type MySpecialType = string & number;

Sometimes you can construct a disjoint type in places such as return type substitution and that is totally fine and normally valid. This warning is only for definitions.

Finding cycles in type aliases

Type aliases are a little complicated because they have to be defined before they have a value (this will be covered in a minute).

Because of this, it can become a problem that a type can be defined in-terms of itself.

type X = X;

// or mututally
type Y = Z;
type Z = Y;
type X = X;

// or mututally
type Y = Z;
type Z = Y;

This is a problem, the type isn't never or any or any other type. It is an invalid construction and not well defined. If it was well defined then it could break.

Thus check disjoint-nessbetween the type alias and its value to catch the issue.

Note we do not have to worry about generics. We can't form this loop type X<T> = T; because the alias always requires an argument. The initial problem equivalent to X<X<X<X<X<...>>>>> which we can't construct.

There are still some edge cases that just checking disjoint-ness doesn't account for.

Reading "properties" of types

The subtype and disjoint operations are great. But other times they don't quite fit into our general type checking. Sometimes we need more manual lower level specific approaches to deal with types.

The first place where we need do some more manual process if with object properties. Because in JavaScript objects are collections of properties, we can get fields from them through access expressions

obj.property
// or
obj["property"]
obj.property
// or
obj["property"]

Depending on the type, we either have quite a straight-forward time calculating this object property (e.g. ({ property: string }).property => string) or quite a hard time.

The following are cases where we need to do some more manual inspection (rather than just disjoint or subtype operators) when working with types

Property access. Covered above.
Getting the callable aspect of type. The details covered later.
Numeric, bitwise, binary and relation operations computation on dependent types
await
instanceof, typeof etc. Easier just to look at the type, rather than do easier

Note that subtyping and disjoint uses the property reading methods to do the subtyping and disjoint conclusion. For property access: we could define property access as subtype with { [*property*]: T } and return whatever the paired value of T is but you can see how this becomes 1. a recursive definition and 2. is a lot more overhead than just looking up the property directly.

Reading properties on conjugations

With intersection types we pick the first result from either the left or the right (none else). With union types we have to check that both types have the property.

Retrieving and calling functions

Similar to retrieving properties, getting the function has to look at conjugations in the same way.

When calling the function we have to handle both value arguments and type arguments. The first step is to handle the type arguments if they exist. In the following call we match them up (like regular parameters), this results in type arguments as [(T, number), (U, string)].

function func<T, U>(a: T, b: U) {}

func<number, string>(1, "hello") // [T = number, U = string]
function func<T, U>(a: T, b: U) {}

func<number, string>(1, "hello") // [T = number, U = string]

After any type arguments, we match up value arguments against regular parameters and check that they valid via subtyping (shown above). If type arguments were missing in the call-site then there is an additional inference step combined into subtyping which will be covered in due course.

Type argument substitution

When calling a function with generics the output type can be based on the type arguments. After all the value and type parameters are dealt with, we need to come up with a type for the result and thus the entire call expression, so that we can find faults in the following.

function id<T>(t: T): T { return t }

// cannot assign string to number
const x: number = id<string>("hello");
// cannot assign "hello" to number
const y: number = id("hello"); // covered later :)
function id<T>(t: T): T { return t }

// cannot assign string to number
const x: number = id<string>("hello");
// cannot assign "hello" to number
const y: number = id("hello"); // covered later :)

To do this we take our domain type T and we walk its structure (in this case quite simple as there is no conjugations) and if find a generic type like T we look in our parameter-argument pair array (which is [(T, string)] in this case) and if any of the first pair items is type-equal (has exactly the same identifier) then we replace the type in the output with the second pair item (in this case the string).

In cases of objects, functions/methods etc we can apply the partially applied generic optimisation so that we don't walk the properties and substitute everything eagerly.

Substitution also happens for property access, for example array1[4] we substitute T as the value for T on the array

The type checking process

Technically type checking is really just the process of doing all the operations above (subtyping, disjoint-ness, property inspection, etc). The process of type checking on a module/file involves lots of small type checks on all the parts such as deep down to single expressions and single declarations.

To do this we walk the AST in a recursive descent sort of way starting at the smallest expressions first. The type checking process is quite involved as we will see so we don't use standard walkers. Instead there is some technical details to walking that will be covered in the next few steps.

It needs to do several passes in blocks
An expression needs to pass down an expected type
An expression needs to return a value type

Contexts

So far we have been talking about bringing together random types. Contexts are the structure to the system, it is what allows us to reference types and reason about expressions in the system.

Contexts store things, the fundamental information is based around two namespaces

Variable names to types for expressions
Named types to types for type annotations

function func(parameter: number) {
	//                   ^^^^^^ get "number" type from the type namespace
	return parameter
	//     ^^^^^^^^^ get type of parameter from the variable namespace
}
function func(parameter: number) {
	//                   ^^^^^^ get "number" type from the type namespace
	return parameter
	//     ^^^^^^^^^ get type of parameter from the variable namespace
}

Rust has a third for macros

Contexts are contained within other contexts. We start off with a root context. Subsequent contexts have a reference to their parent context. Once a scope has been exited the contents of the namespaces are dropped as they are no longer needed (unless stored for LSP work).

Sometimes a item can have a variable representation and a type representation (for example class has a general prototype type and a variable constructor type)

class MyClass {
	constructor(n: number) { this.n = n }
}

const x: MyClass = new MyClass()
//       ^^^^^^^                 used as a prototype type equivalent to { n: number }
//                     ^^^^^^^   used as a value to mark the constructor
class MyClass {
	constructor(n: number) { this.n = n }
}

const x: MyClass = new MyClass()
//       ^^^^^^^                 used as a prototype type equivalent to { n: number }
//                     ^^^^^^^   used as a value to mark the constructor

This comes with the confusing thing that the variable MyClass is not the type of MyClass. My class the expression when used produces objects of type MyClass when constructed using the new C(values) syntax.

Types vs type annotations

Every term has a type. We can add notations for types.

By convention many language instantiate parameters using chevrons <> rather than parenthesis like regular function calls. Golang added (square) brackets. Lean has consistent syntax for both.

Languages that have chevron groupings for <> generic arguments and >> for the shift operator can create havoc for parsers that are not completely context aware.

Every term has a type that can be resolved from the context. Type annotations allow places to add type information (in the form of constraints).

Slightly different to type annotations, there are also type declarations in the form of aliases and interfaces.

These annotations behave differently to normal syntax in that (for the most part) they just denote information to the compiler. They can be totally erased (or replaced with whitespace).

It is the sort of thing the + operator will end up in the output as something like addsd.. But you won't normally be able to find the origins of the types of data.

Reflection can change this. Things like enum have runtime meaning so there are edge cases where types

Gradual typing with elided annotations

Type annotations assign types to variables, parameters and function return types.

In languages that are supersets of a language that doesn't have types. Also sometimes types can become verbose to write and not applicable for non-foundation work (demos, experiments etc) and so systems which don't require any annotations.

any is used as the type for missing annotations. Because of this it simultaneously acts as never and any in the subtyping in an attempt to make it easier to adopt in codebases that don't have type annotations already.

I think parameter constraint inference could be a lot better in languages. The Hegel checker had a cool approach to this and there is some WIP things coming to ezno-checker, where we can be a lot smarter about missing annotations

Inference

The inference process normally means forming a type based in its context.

Inferred variable constraints

In the following there is no constraint on variable a. We could say that it type is "hello" but that would break the later reassignment, therefore we instead chose the basis of "hello" which is the string type.

let a = "hello";
a = "world"
let a = "hello";
a = "world"

Because there are no nominal types, inferring this restrictions can become a little bit complicated. For example while totally valid JavaScript a = 2 would not be allowed because of the inferred constraint.

In Ezno the variable constraint and values have a special implementation thanks to aware-ness of re-assignments in side-effects. So this sort of "inference" doesn't have to be implemented. From this the system can just say that variables can be reassigned to any and can be more flexible in cases (that you should probably not create anyway). However in some cases simpler to and more realistic (for example for (let x = 0; ...))

Inferred type arguments

While functions can have generics parameters, we don't require them to be specified at call-sites. Show above before but we can have.

function identity<T>(t: T): T { return t }

const x: number = identity(5);
function identity<T>(t: T): T { return t }

const x: number = identity(5);

He we infer that the T type parameter is a number. This is done during subtyping, when we check T <=: 5 we see that T does not have a value yet and so pick on out based on our RHS type 5.

TSC has a two pass system first the bind phase. In Ezno I combined the two, merging it into the subtyping.

More on how this works for in other cases in a future blog post.

Inferred function parameter types

When we do the checking stage scanning expressions and other AST, we can "pass" down a expected type. This allows for adding types to parameters without them.

Return type "inference"

Similar to variable reassignment, we might also not have a return type for a function. There is probably many ways to handle this but in ezno-checker, a return type is picked based on control flow analysis when checking the inner block

function func()     {
    //          ^^^ no return annotation here
	return 256
//  ^^^^^^ ^^^ returns `256` so we will say that number is the return type
}
// func: () => number
function func()     {
    //          ^^^ no return annotation here
	return 256
//  ^^^^^^ ^^^ returns `256` so we will say that number is the return type
}
// func: () => number

Actually in Ezno a "second" return type is always built under a slightly different mechanism. So you always sort of infer the return type. You can try that out here.

Reflection and narrowing / refining a type

I will do a whole blog post on my approach to narrowing and more but I will give a brief overview.

In JavaScript there are various expression level operations that allow us to inspect a type such as typeof and instanceof. Because there is no pattern matching (👀) the way to do things conditionally based on types is using these expressions.

Inspection of values for type information is known as reflection. You can see the wording when you think of a program telling you about itself.

function func(param: any) {
	if (typeof param === "string") {
	//  ^^^^^^^^^^^^^^^^^^^^^^^^^ based on this expression
		const x: string = param;
		//                ^^^^^ we decide this is of type string 
		//                       inside this conditional block
	}
}
function func(param: any) {
	if (typeof param === "string") {
	//  ^^^^^^^^^^^^^^^^^^^^^^^^^ based on this expression
		const x: string = param;
		//                ^^^^^ we decide this is of type string 
		//                       inside this conditional block
	}
}

The full approach involves build union types when the expressions involve logical or s || and unions when looking at logical ands &&. Things get slightly more complicated with negation and predicate functions, so I will leave that for a future blog post.

Representing types

So we know we have seen how we might operate on types. But one of the hardest things for me what figuring out how to represent these in a checker. Writing the checker in Rust was initially challenging because of the constraints. Here are a few things about types that should be considered.

Types can be cyclic

Types can be recursive they can have properties that represents themselves. This proves difficult in Rust because the non-mark sweep garbage collector approach puts limits on what can be represented.

While I call it an Arena, its just a Vec

Initially I thought just a Rc would work, but that can run into deadlocks where a type which references itself somehow cannot ever be freed because there is cycle in the reference. While not an issue in the CLI, in the situation of a LSP (the checking done in an editor) which stays around for a while, this memory leaking could crash the LSP over time

Arenas also are a good fit here because the types stay around for a while after passes. We know when they are no longer used and we can do the clean-up manually. Individual counting would amount to a lot of increments and decrements here and there and would be too fine grained of an operation. If I do ever get around to a more efficient store for types, it will also be linked to the module and based on the lifetime of that.

Short-circuiting and continuing on errors

One consideration with type checking is around short-circuiting.

For example "type checking" the current file we find a issue with the first variable declaration

function func(): number {
	const variable: string = 4;
	const something: number = "Hello World";
	return variable
}
function func(): number {
	const variable: string = 4;
	const something: number = "Hello World";
	return variable
}

Now one way is to walk the tree and on the first issue we find we return TypeError. But that would be pretty annoying to use this checker. We fix the issue (in this case either the RHS expression is incorrect or maybe our annotation isn't correct for this situation) and then re-run the checker to see it is fixed. This now shows a second issue with the const something declaration.

It would have been nice if this issue could have be presented to us on the first run. So we can improve the checker by rather than returning an error, we append to a list of errors. So now on the first run we get three errors.

There is a decision to be made here about the type of const variable here? Do we make it of type string or of type 4 (aka number). We could not define it at all but that seems invalid as the variable is defined.

Here I decided to treat the annotation : string as the type const variable has (with a marker on the type to say it is an error).

But there are cases when we can't rely on the annotation.

const x: NonExistantType = 2;
const x: NonExistantType = 2;

Here we take up unknown annotation as any (with a marker)

but there are other times where there isn't an annotation at all

const obj = { prop: 1 }
const variable = obj.property;
//                   ^^^^^^^^ "property" does not exist on { "prop": number }
const obj = { prop: 1 }
const variable = obj.property;
//                   ^^^^^^^^ "property" does not exist on { "prop": number }

Here we take up the value is never (with a marker)

This is also a tricky problem for syntax parsing. If we encounter syntax that doesn't look right such as function func() { = if }, do we parse the RHS as only an expression or as full statements? There is a trade-off in flooding the user with a block of errors some of which might be dependent on a original step VS the original problem of only surfacing. Some of this can be negated by one-syntax error per file etc.

While it might be good in CI to fail fast, when you are working live in an editor it is often common that some syntax might be incorrect. This handling of partially valid sources is needed for autocomplete etc. We actually need an AST to check before we can do this let x = ;. Will be covered in a post all about parsing. Syntax errors are much more of a problem than with types.

These set of features is current under the term "stability".

Types vs values and unknown-ness

Dependent types

Dependent types were added to TSC because JavaScript does not have a specific structure for tagged unions of data. Instead a so a lot of existing APIs used these "tagging" pattern to discern between objects.

declare function getUserActivity(): Array<
	| { kind: "follow", from: ..., to: ... }
	| { kind: "comment", on: ..., comment: ... }
	| { kind: "post", id: ..., caption: ... }
>;

const activity: ...
if (activity.kind === "follow") {
	// ...
} else if (activity.kind === "comment") {
	// ...
} else if (activity.kind === "post") {
	// ...
}
declare function getUserActivity(): Array<
	| { kind: "follow", from: ..., to: ... }
	| { kind: "comment", on: ..., comment: ... }
	| { kind: "post", id: ..., caption: ... }
>;

const activity: ...
if (activity.kind === "follow") {
	// ...
} else if (activity.kind === "comment") {
	// ...
} else if (activity.kind === "post") {
	// ...
}

This pattern is a lot easier to understand than using from in activity checks.

You can also use unions of classes with instanceof. But functions such as JSON.parse and Response.prototype.json return objects without a prototype.

`5` as a type

In a sense 5 as a type is the number type with a identity type from itself to the term 5. For simplicity sake and because ezno-checker is a not based on something that supports propositions-as-types is simpler just to define Constant as a variant of Type and say that similar to number having properties, the type 5 is something with all the behaviour/properties you would expect for a 5 term.

`const` systems

In some systems they allow dependent types only in certain places. Because in Rust, generics must extend specifically trait types, there is a special const modifier for parameters that reference types as the extends constraints instead so that these generic parameters expect arguments to be values rather than types. In Rust const parameters are used to set the number of lanes in its upcoming SIMD API. While you could create a struct Eight; etc. Using this const version means that you can use operators.

In TypeScript there is also the const modifier for generic parameters which tells TypeScript to handle the arguments a little bit more tightly in these cases.

The Zig language also has some interesting ways of handling types at compile time that I unfortunately didn't have the time to look at for this post. If you have some interesting posts about Zigs type system features (especially around comptime) leave them in the comments.

Wrapping up

And that is all for now, hopefully you enjoyed this mostly high level overview of types and found some of the subtyping rules interesting.

If you liked the diagrams interactive or not, let me know whether I should do them in the future?

I would like to talk about the Curry-Howard isomorphism and Rust lifetime types in the future which are the more advanced parts of type theory. I also have left over material on reflection and the taking a more interpretation/computational based approach to type checking. If either of those sound interesting you can thumbs up the comments and I will add it in.

This post was suggested by one of my sponsors. If you want to read drafts of upcoming blog posts you can sponsor for $5 a month.

I am also now on Bluesky, if that interests you can follow on there.

This is the first in the trilogy of posts deep into type checking. In the next posts I will build on some of this with more on the advanced parts of TypeScript as well as some things I have been working on in my own approach for a type system. So stay tuned!

The quest continues

2024-05-31T00:00:00Z

It has been nearly a year since the last post so thought it would be good to do a much-needed status of the project! While not production-ready yet there has been some great recent progress in the last year. The project has passed 2000 stars, 10 external contributions and it now passes over 200 checker tests.

If you haven't heard of it yet, Ezno is a new TypeScript type checker that experiments with deeper analysis to raise new errors and find new optimisation opportunities!

This post will go over more of the state of the compiler and tooling. While there has been some huge technical progress on the checking and compilation side, I will leave it out of this post and split the technical details across future blog posts!

There is a new web-based playground

This can be used to try the checker. You can see some of my favourite features here

Note the playground is WIP. Identifiers for sharing currently use a sequential index, which may change in the future.

You can also try the checker locally with the CLI from the latest GH release

The test suite

Since last year, I have started on a new collection of type-checking tests for the checker. In its latest release, the checker passes 235 tests!

There have been a lot of improvements in TypeScript features: complex subtyping, mapped types and extends annotations. There is now a good foundation for the checker but there is a still a lot to do. There are some general JavaScript and TypeScript features to finish:

Inference (was working, now being redone to work around Rust's mutability)
Narrowing (got some parts working, working on getting more edge cases narrowed)
More collection logic (Array, Map, Set etc) (array methods have been added, but are yet to work perfectly)
Generators (require storing and partially evaluating events)
For-of (should work by de-sugaring iterator calls, but a bit complex)
Recursion
Functioning queuing/non-synchronous calls and promises (requires more unknown-ness and coordination between events)
Improving how types are stored (current implementation doesn't scale for larger projects), which should then allow for more of the standard library to be used
node_modules imports (files work, zipping source up with definition files is left to do)

and there is a general need to improve all the different edge cases in which break the checker.

There is no time frame or deadline for these. But as long as I can continue with the current trajectory they will eventually be all good!

At the moment there are 235 passing tests. The hope is that the checker should be ready for simple projects at around 300 passing tests and larger projects at around 500 passing tests. You can see a collection of things that do not work in to_implement.md.

I am looking for more contributors to the checker tests. If you have a favourite TypeScript or JavaScript feature not included in the current specification (or to_implement.md) it would be really helpful to contribute it

These tests are authored as markdown and automatically built into tests that Rust can run. You can find out how a markdown file works as a test suite here.

New additions from contributors

Thanks to Charles Taylor for adding default value checking and checking for the type of catch variables.

Thanks to Lemuel DLS for checking the edge case for destructuring with this

Thanks to Patrick Laflamme for adding excess property checking (including across conditionals which TS does not check currently as was pointed out in a this issue)

and again for setters

Thanks to Apika Luca for adding import.meta

Additionally, there have been improvements internally to the project. Wan added positions to events which have enabled emitting some new errors (that will be featured in the future). Jules Guesnon added export type parsing and improved the clippy configuration, which has helped keep things in order as the project grows. There are now fuzzing tests for the parser thanks to Caleb Jasik and Addison Crump, which has caught some problems in parser and to string-ing (turning the AST back to a string). Nathan added the parsing for try-catch-finally. markthree added CJS support for the JS library and Nicholas Roberts added type definition generation for the JS library (the type definitions have been very useful when building the playground). You can see these contributions and more here.

Despite Rust being a non-trivial language to author and the type checker being very complicated and entangled, I have been impressed at the contributions to the type checker and project!

Bad news

Unfortunately, the first half of the year has been a bad month for JavaScript type checkers. First, the STC (speedy type checker) was abandoned and now Hegel is over. As someone who knows the difficulties of building a type checker, I commend the incredible work put into the two. Hegel is especially impressive with its parameter constraint inference, which is an incredible feature I am hoping to implement as well in the checker.

I am hopeful that Ezno's wider feature scope compared to STC is more of a safe option (Ezno is not exclusively focused on checking performance). While I thought Hegel made some great trade-offs to provide better checking it seemed maybe too much for potential users compared to TSC. I will explain more of how I think you can add better checking behavior while not breaking existing valid cases allowed by TSC in future blog posts!

Also, the ezno-checker is no longer included under Oxc and the only current way to use Ezno is through its own toolchain. Oxc is an impressive project for performance but differences in timelines and targets meant that it didn't make sense to put the work in maintaining the AST bindings. That is just for the moment though, the checker crate still keeps the checking logic separate from the AST/syntax implementation. Hopefully in the future when more of the language is implemented other tooling can consider reusing or copying parts!

Supporting the project

I have written some good first issues in the issue tracker. If there is something you want to contribute then I can help explain certain nuances in the checker and give steps to start! You also don't need to be good at Rust to contribute, adding more checking tests is useful for knowing how to approach new features! Type checkers are hard and Rust is not an easy language to write in. However, at this stage, you can have a really big impact by adding fundamental features! I appreciate all pull requests no matter how ready they are (I would rather have some code that doesn't quite work than none at all).

I have GitHub sponsors, which very generous people have signed up for! I have uploaded the LSP (a plugin for doing type checking in editors like vscode) to a sponsors-only repository and plan to also commit other experiments and early blog posts there. That is available at any level of sponsorship and I also plan to add code contributors to the main repository there as well!

Unfortunately, a lot has come up in the last year which is why it has been a while since the last update. Again, a lot of progress has been made internally so I hopefully can share some amazing progress on type-checking and compilation soon!

Cool things in '23

2023-12-20T00:00:00Z

I thought a good way to see the year out would be to show some appreciation for some tools and projects that I think made the good list this year as well as some things I am looking for and forward to next year.

Tools

Figma

Been using Figma as my main graphics editor for a little over a year now. Here are some things I think are well designed:

Components for composition
Simple controls (such as k for scaling, space for moving)
The pencil drawing tool. Although I wish you could define the number of points it creates, I often want less vertices and detail and often get lag when there are a lot on the page.
Auto layout options
The outline/layers/elements panel. I like you can modify individual objects in unions and intersections.
Plugins such as lorem-ipsum, swap, quick-tracing and for more advanced things scripter
The free tier is enough to have infinite pages

I also have been learning more about at pen/Bézier tool. Did you know you can press ctrl to break out of a spline and also click on the node to get out of curvy mode/make one-sided influence.

Also a tip here: if you are exporting a design to After Effects. As After Effects can't import SVGs you can instead export a EPS using this plugin and then import that into After Effects and break into a shape layer to keep the vector-ness of the graphics.

Obsidian

Obsidian is a markdown editor. It is what I am currently using to write what you are currently reading. I will skip to the key things I like

LaTeX support. As someone who writes lots of mathematics notes it is great that I can do $\Gamma (z)=\int _{0}^{\infty }t^{z-1}e^{-t}{\text{ d}}t$ and it renders inline in the editor as
Plugins and theming support
- LaTeX shortcut plugin is a must-have for writing mathematical formulas. I don't think I can go back to writing latex character by character. My snippets are here
- Zen mode is also great. Using Zen mode and full-screen seems to be the best non-distraction mode for writing
- Surfing extension is also a great way to find things without tabbing between windows
- One of the benefits of it being built using Electron is the ability to easily add custom CSS for the UI. My custom darker-than-dark CSS is here
Canvas is nice for structuring ideas (I need to use this feature more). Although I normally find nested lists to be the best way to get ideas out of my brain
Exporting to PDF (unfortunately no dark mode export)
Split editors/panels are great
Mobile app (although debugging Obsidian plug-ins on mobile is impossible)
I always leave a couple of TODOs tags in each document. It is nice it highlights it and can search it easily across documents. Although it really annoys me that an open parenthesis followed by hashtag doesn't create a tag. Hopefully, despite being brought up years ago, it will be improved in the future
Viewing PDFs in the editor is incredibly useful. Although I wish it could natively render in dark mode (figures in white, background in black). I have tried with custom CSS with filter: invert(1); it works, but I get weird/annoying chromatic aberration (see image below) and has the comedic but slightly annoying effect of inverting any images. Also had issues with scrolling PDFs losing their position and going into a weird state when the native one first came out. Hopefully, that has been fixed

Problems I have in Obsidian are

Opening files can close a file I was working on. I wish it worked the same way as vscode
Saving every keystroke. It makes my laptop fan spin up to max if I write files while the 11ty live reloading is on. Wish there was an option to add a timeout from 10ms-60s.
Some way to split up a file by headings. I like big files, I don't want them rendered all at once.
Sometimes slow, especially when it is the 3rd chromium based editor open

So when I don't use Obsidian to edit I use

Helix

Helix is a code and text editor. It works in the command line similar to vim and emacs. Things I like

Simple to set up, seems like most things work out of the box or are connected automatically
Controls are a little opinionated but in a good way (although I still have to look up most of the advanced ones)
Line wrapping
Lots of (good) themes
Split editors are good
Finding in a file is good (I wish the global one was a little better and I really want a better global search and replace as I use that in vscode a lot)
Multiple cursors in a column (C or shift+c ) and extending selection (* to v to n) is good (what in vscode is ctrl+d). Although I have had a few times where I also had another cursor down the page which I didn't realise (and clear cursors with ,) and ended up messing up a bunch of things offscreen
The editor can be a bit slow sometimes, keystrokes lag in larger files. I think it might have to do with syntax highlighting. They recently improved fuzzing when searching which I found a lot snappier afterwards

Some improvements I would like to see in Helix

Don't show images in the file picker. It also doesn't show files in .gitignored paths which is annoying
I don't use the LSP integrations, as I find the placement of the diagnostics a little strange and sometimes slow. For quick edits to files and just drafting things it is really useful though

Rust

I have written a lot of Rust this year across a number of projects. What has been amazing to see is some of the minor crates I have built being used in other people's projects. Somehow the total downloads across my crates are now over 200k. Will be interesting to see the stats next year.

I also recently received a PR that enabled clippy lints. I hadn't tried clippy before, beyond its default mode. The PR tidied up a lot of Ezno's internal code and also found a bug. If you use Rust you should check out the page that lists all of the clippy lints and consider turning on more by default.

Rust's online playground is still golden. It is a great way to start as a beginner or to test whether something works without spinning up a whole project. The option to build on nightly and do formatting are nice touches.

Lean

A slightly niche one. Excited to see Lean 4 released this year. A really interesting language, and when I don't get too try-hard about code golfing every proof it can be really fun. The main manual is okay, but the theorem-proving and functional programming books go into much better detail. Would love to see an official blog (similar to Rust the release blog) to stay up to date with its development.

Mathlib, a huge project aimed at formalising a huge amount of mathematics is now on Lean 4. While it is a great foundation for proving things. I wish there was more intermediate material and codebases. I played around with Lean 3 a while ago and I have just open-sourced some things I wrote. You are only allowed to make fun of my proofs if you also put in a PR to shorten them 😆😉.

I also think the OpenAI's usage of Lean is incredible interesting. Mixing the output of AI with the formalisation foundations of type theory for validating output might be a big thing in the future. Anyway hopefully ChatGPT's cut off date gets sooner so that I can use it more for the newer syntax and features that arrived in the fourth version of Lean rather than from the third version.

DuckDB

Earlier in the year I built a Python application with predictive text. I used the DuckDB CLI to do some modifications to a CSV file that I took from a Kaggle (my updated fork is here). Although it was a medium size dataset, running the data commands felt instant. Seems like a really good way to play with SQL on the client, it has better documentation than I could find compared to SQLite and has some useful internal functions.

GitHub actions

The great thing about GitHub repositories is the single public source of truth. One of the troubles I had before using actions was deploying. Before, I had to get the latest repo, deploy from my own machine (checking I had all the keys/credentials) and then commit with the updated version and state.

With GitHub actions, on all my Rust projects I have set up a workflow for deploying crates (crates = Rust packages) that not only deploys specific crates with certain versions given as inputs but also automates the tagging and committing of updated manifests. Any pull requests to the projects run lint checks and tests so I can see whether they are valid without cloning the changes locally (I think I have now perfected my Rust setup).

The other good thing about Repositories being centralised is benchmarking. If you do publish benchmarks it should be required that the methodology is public. With GitHub actions not only can you make the testing script public but also the exact logs and run using actions. I now have a benchmarking repository that is all automated so anyone can go and test out benchmarks for themselves and PR to make corrections and improvements.

One feature I want is: it would be nice if there was a way to have actions in a central place. Every time I set up a Rust repository I use the same workflows. Rather than having the same bunch of files across repositories. It would be nice if you could do something like this in a .github/workflows.yml.

include:
	- "gist file here"
	- "maybe a repository link"
	- "someway to include a bunch of workflows on the main branch here"
include:
	- "gist file here"
	- "maybe a repository link"
	- "someway to include a bunch of workflows on the main branch here"

A tip for all those having difficulties wrangling GH actions: Create a private repository for testing and just go wild on there. By trying certain workflows and wrangling yaml syntax on there I have mostly avoided creating clutter on main repositories.

On the command line

Here are some other tools that I use (when I remember what their parameters are)

SCC counts source code lines. Other than some silly statistics by default, it is great and I use it (with its JSON output) to keep track of how many lines I write in projects
bat prints the code for files with highlighting. I wish -pp was the default for <40 LOC files, but it's quite easy to remember 😆
JQ, a JSON processor, selector and transformer. The playground for it is great
FX, a JSON viewer
GitHub CLI. I always get mixed up between the run and workflows commands, but apart from that very good

JavaScript and the web

Natto

Sometimes I just want to do a quick calculation. Maybe split some text and get the fifth line etc. Then sometimes I realise I now want that to be uppercase. Later, I might come back and depending on where I saved it have the following problems

A shell session: where if it was saved, I now have to trawl through a bunch of unorganised lines and I can't edit functions because that isn't how REPLs work
A file: which is sometimes easy to lose and then often hits the weird limit of being a file too large for the screen that is a mess of logic. Then I have to start up a runtime in watch mode and remember to log everything
An online service (Replit, Codepen, etc): which can often be slow and often tricky to do things locally for privacy reasons (wouldn't want to accidentally copy an access token out to the world)

Now instead of any of the above, I always go for Natto. Natto has a unique design in that all the code is contained in blocks that have outputs that can be connected to other blocks as inputs. I have found that this node-based design is an ideal way to express demos and small pieces of code that are beyond the one expression mark, but before the point when it starts requiring sub-directories, a git repo and a linter.

Here is a fun demo (which you can even try here) I made where I used Natto to structure something I was experimenting with

It has a nice interface (despite no dark mode, which should be a minimum for developer tools after the advent of CSS variables) and it is always instant to start. You can develop both private local nattos (which are saved as JSON files) as well as the option to publish them online. For one-offs, the default evaluation block prints the expression (rather than having to remember to console.log it). It also can render HTML directly which is great.

However, I have problems with the following

I can't figure out a way to do timed iterations/run code at intervals. I think Natto is all built off React, so when using non-pure things like setTimeout, I need to add special clean-up logic
I wish there was an easier way to switch evaluation block type (aka just type /functionbody/html in the editor)
console.log goes off to the left of the screen when evaluation blocks are fully windowed

Unfortunately I don't think it is maintained anymore :/

Val Town

"Val Town is a social website to write, run, and deploy code". Similar to Natto, ValTown is a great playground for JavaScript code. Unlike Natto, its playground code can be deployed to servers to run on a schedule or a HTTP endpoint. I think it is the easiest and quickest way to get server-side code up and running.

A while back I wrote some simple automation scripts that would generate and store SVGs after being called by a HTTP endpoint. It took a while to find a free service that could also store data and then I had to create a project locally and deal with bundling for my <50 LOC scripts. I now have no idea where those are on my filesystem, when they are run etc. Looking at ValTown it seems like it would have been perfect for hosting a bunch of my small automation scripts. It has a generous free tier, KV storage, logs and best of all is in this centralised store of code which can be made public. With lots of features dropping I am interested to see where it goes!

There are some really interesting examples and use cases and you can also try Ezno's checking API (which uses WASM behind the scenes).

WASM

Web ASseMbly is a low-level format that can be run in various runtimes including JavaScript on the server and browser. Despite Ezno being written in Rust it can run in the browser and server runtimes thanks to the Rust compiler supporting WASM as a target and wasm-bindgen.

WAT (WAsm Text representation) is a great way to play with the WASM format but I have also tried out wasmati a DSL-like library for writing web assembly and it's pretty good. WASI is also an interesting development. It adds native system calls to the runtime, which if enables the reduction of language-specific glue code might be great.

Next year I would like to see the component proposal make some progress. I would like to see more exploration of how to make a WASM module that can manipulate strings in the host language without the overhead of copying memory in and out of the sandbox. And maybe I am being naive here, but I wonder whether there could be a WASM to machine code compiler, as I think the runtimes are at most JIT rather than AOT. I might be wrong about this, so feel free to correct me in the comments at the bottom of the page.

Cloudflare web analytics

I use Cloudflare web analytics on this site. Numbers shouldn't matter, but if it is possible I would like to know whether my blog got 5 views of 5000. Unfortunately, the web portal is a pain as it requires 2FA every time (I wish it was only every two weeks), so I normally just read the email notifications (which have the benefit of being stored forever rather than expiring after three months).

Other cool things for the web

HydraActive looks incredibly cool. A slightly different attempt to what I attempted a few years ago. Seems like a much simpler way to do what I attempted with less compilation and static analysis using what seems to be signals!
Million.js: Looks like a great and simple library to implement in existing React applications that run on the client. Maybe one day VDOM will be over, but for now this looks to be a great way to speed up some updates in React land and since it is only a plugin it should be easy to add without big changes to existing code
Wing a programming language orientated for cloud specific things looks cool. I wish there were more experimental languages experimenting with things other than syntax, logic and runtime
additionally unison is another language that has interesting effect detection
Arktype looks cool. Maybe the best implementation of types at runtime
Oxc toolchain. Some incredible engineering has been put into the parser
The giscus comments box (despite being a next app in an iframe, which I unfortunately have to grit my teeth through). I found it incredibly simple to add to this site
A few standards for JavaScript I have seen that I hope could simplify development
- pattern matching IF AND ONLY IF IT IS BASED ON THIS
- defer import eval hopefully we can get rid of dynamic imports (at least the non-static ones that are ridiculous)
- and type annotations if they don't include satisfies. Although I think there are lots of alternative approaches to reducing the problems from type annotations, if it means publishing code with type annotations to NPM is possible, then that would be great.
11ty still powers this blog, haven't had any problems with it
and Jampack optimises the output. A great SSG agnostic optimiser!
Leptos is a reactive rendering library written in Rust. Looks like a great way to build client applications using all of the language and tooling of Rust
Shuttle is still the best way to start deploying Rust applications
With a new build tool for JS written every week now. unplugin is a great abstraction for building transformers and other tools for JS that can be used (almost) everywhere.

Things I am on the lookout for

If you can find these (or make them), leave them in the comments below 👍

A terminal that renders markdown. More than a notebook. VS Code added images, I need more such as italic and bold decorations, links, code blocks, tables and for the whole nine yards LaTeX rendering.
I would like an email function as a service. Similar to an edge function with a HTTP endpoint. I want one that can email and it replies to that email with some information.
Maybe shouting into the void but I still can't grasp how for iOS alarms there isn't a mode with doesn't make sounds and excessively vibrate. How come such a simple feature doesn't exist??? Keep having to check the clock every five minutes in libraries.
I am always using the browser mini-player. It unfortunately only includes controls to pause with no feature for going back and forward ten seconds :(
I also want GitHub for databases. As mentioned above I uploaded a dictionary dataset to Kaggle but there doesn't seem to be a way to make pull requests on it to add, fix or remove data. Maybe HuggingFace might be the place for that but I am not sure. GitHub doesn't seem like the best place and doesn't have the same interactive tables that Kaggle and HuggingFace do.
I also really want a GitHub action/bot to apply clippy --fix and rustfmt format on the current PR. I don't know how I feel asking the author to do the sort of bookkeeping in a project if simple checks fail. It would be nice if I could comment @my-bot try fix lints and formatting and it runs it and commits the fixes (rather than me having to checkout and do it manually). Is this even a good idea, does it change who authored it when it appears in git blame`? If you know a simple solution feel free to PR it.
Artificial intelligence has been a big thing this year, although I haven't observed anything cool wasn't from the previous year. I still really like: Dalle (they should give more free credits out now though) for generating artwork and ChatGTP is occasionally useful for finding synonyms of phrases and shell scripts but still isn't perfect. I would like an AI that can find songs from snippets and one to generate the one minute instrumentals from snippets would be amazing 🙏. I wish that more of the LLM chatbots were more up-to-date. They continue to be useless for anything that has happened in the last 18 months.

And that's it, to 2024 🎆🎇. If you worked on anything cool or something leave it in the comments!

A preview of Ezno's checker

2023-06-23T00:00:00Z

The big news

Ezno's checker has now been open-sourced!

This post is an overview of the current status of the project and goes into more detail about its unique approach to type-checking!

The aims/direction of the project

Two things:

It is not aimed at having one-to-one parity with TSC. I made the point last year that TypeScript's implementation of any (and its other backdoor behaviours) make doing type-based optimisations and complete type safety impossible. In my opinion, 90% of TSC features' are great, so the aim is to be able to use existing syntax and not to break things in a bad ways.
Following on from 1) this is an experimental project. I want to explore new techniques and make previously impossible things possible! Building it from scratch makes it a lot more fun and easier to build for me.

There is still a lot to do, a lot of JS features are still unimplemented. I am only human :smile:, while I can implement features quickly, some other things will require time (also not to overwhelm myself).

Checking performance

Don't want to dwell too much, there are a lot of places where Ezno could be faster I just haven't had the time to improve and the project is not at a stable place to work on optimising current behaviour. However I have seen a few comments that "Ezno does more, so will be slower", so I did some investigating and have some pretty interesting numbers:

Benchmark 1: oxidation-compiler check ./demo.ts
  Time (mean ± σ):      48.5 ms ±   1.4 ms    [User: 33.6 ms, System: 15.0 ms]
  Range (min … max):    46.6 ms …  55.4 ms    60 runs

Benchmark 2: tsc --pretty demo.ts
  Time (mean ± σ):      1.658 s ±  0.165 s    [User: 2.364 s, System: 0.103 s]
  Range (min … max):    1.542 s …  2.110 s    10 runs

Summary
  oxidation-compiler check ./demo.ts ran
   34.21 ± 3.55 times faster than tsc --pretty demo.ts
Benchmark 1: oxidation-compiler check ./demo.ts
  Time (mean ± σ):      48.5 ms ±   1.4 ms    [User: 33.6 ms, System: 15.0 ms]
  Range (min … max):    46.6 ms …  55.4 ms    60 runs

Benchmark 2: tsc --pretty demo.ts
  Time (mean ± σ):      1.658 s ±  0.165 s    [User: 2.364 s, System: 0.103 s]
  Range (min … max):    1.542 s …  2.110 s    10 runs

Summary
  oxidation-compiler check ./demo.ts ran
   34.21 ± 3.55 times faster than tsc --pretty demo.ts

These are rough benchmarks, so take these numbers with a grain of salt, they could go up or down. You can see the full result in my fancy automated benchmarking repository

STC: Speedy type checker

Ezno is not the only TS checker written in Rust. STC (which was made public last year) is aimed at following TSC one-to-one, with the aim of improving the type checking time compared to TSC. Similar to Ezno in still being work in progress. Its current results look very impressive, being slightly faster than Ezno with Oxc.

./stc/target/release/stc test demo.ts ran
    2.72 ± 0.11 times faster than oxidation-compiler check ./demo.ts
    89.80 ± 3.56 times faster than tsc --pretty demo.ts
./stc/target/release/stc test demo.ts ran
    2.72 ± 0.11 times faster than oxidation-compiler check ./demo.ts
    89.80 ± 3.56 times faster than tsc --pretty demo.ts

Architecture of a type checker

A compiler is typically a chain of the following processes

lexing -> parsing -> type synthesis & checking -> output
lexing -> parsing -> type synthesis & checking -> output

Ezno's current approach type synthesis & checking is a bit different from others, so here is some more insight.

Synthesis: a bridge between the system and the language

The main ezno-checker system contains high-level memory of the program (it's sort of its heap), it stores all the information about values and structures (encoded as types) and what functions and blocks do (encoded as events). This is all currently done in the ezno-checker crate.

Synthesis on the other hand is taking AST and forming types from it.

Due to some issues with Ezno's own parser and after an offer to help from Boshen I figured it would be a good idea to make the synthesis work under other AST crates. To do this required some changes to split the checker logic from AST synthesis and the creation of a new bridge crate between Oxc AST and Ezno checker.

The bridge crate consists of things like code that turns oxc_ast::ast::Expression into a type

pub(crate) fn synthesize_expression<T: FSResolver>(
    expr: &ast::Expression,
    environment: &mut Environment,
    checking_data: &mut CheckingData<T>,
) -> TypeId {
    let instance = match expr {
        ast::Expression::BooleanLiteral(boolean) => {
            return checking_data
                .types
	            .new_constant_type(ezno_checker::Constant::Boolean(boolean.value));
        }
        ast::Expression::NullLiteral(_) => return TypeId::NULL_TYPE,
        ...
pub(crate) fn synthesize_expression<T: FSResolver>(
    expr: &ast::Expression,
    environment: &mut Environment,
    checking_data: &mut CheckingData<T>,
) -> TypeId {
    let instance = match expr {
        ast::Expression::BooleanLiteral(boolean) => {
            return checking_data
                .types
	            .new_constant_type(ezno_checker::Constant::Boolean(boolean.value));
        }
        ast::Expression::NullLiteral(_) => return TypeId::NULL_TYPE,
        ...

Thanks to this, a check command was added to Oxc CLI and type checking was added to the playground.

The playground is currently work in progress as I only added support in an a hour, expect some crashes that will show up in the browser console

As Oxc has an incredibly fast and correct parser it will be a great way to use Ezno's checker!

This isn't the end of Ezno's own parser and CLI. It was incredibly useful building it, and it's 95% of the way there. It is great that the bindings exist with Oxc to offer a fast and reliable option to use Ezno's checker.

An interpreter-based checker

Ezno tackles type-checking as evaluator/interpreter. However, unlike a real JS interpreter rather than passing around values it passes around types. These are instead spaces that values live in.

Evaluation

Type-based evaluation

Some big differences between what a value-based interpreter would be and Ezno's evaluator-based type checker:

Control flow has to handle different cases. In standard evaluation branches are binary, they either run or they don't run. In Ezno's checker, it's non-deterministic. (unless it can find the branching condition to be true) It has to evaluate both branches and join the results together. Similarly, loops (and recursion) have undecidability to them and have to unify the results to reflect the possible amount of times the block can run.
IO functions don't run. It doesn't make fetch requests or Math.random() at compile time, and it certainly doesn't ask the compiler's user to answer to prompt. But all other functions whether from the runtime or user code are evaluated in some kind. It does the evaluations (most of the time) based on the body of a function, rather than a return type annotation.

The other big difference between an interpreter is that it type checks annotations. While there is no rule that interpreter cannot do this, it doesn't hugely make sense

This approach has benefits for languages with dynamic features. However, this approach still has benefits for stricter languages.

I have a longer, more theoretical post on some additional points, which I will get around to finishing at some point 😀

Less theory, let's see how it works. The next part goes over some code that is used in the current demo. You can try the above first to see how it is implementation works.

Inside the checker and its features

Note that a lot of this is a work in progress and things might have changed after the publishing of this post.

Contexts

Contexts store local information.

This includes:

Names to types
Names to variables
The current value of a variable
The current properties on objects

They also contain a reference to the parent. Which it can get properties from:

What kinds of contexts are there?

Root
Module
Functions
If and else blocks
For statements

Every environment (other than root) has a field for the Scope. Scopes differentiate environments that are for a function body between if-else blocks (which we will see different behaviour for later).

It can get slightly more complex as it (will) include

function (input: MyObject | null) {
	return input && otherValue
	//              ^^^^^^^^^^
}
function (input: MyObject | null) {
	return input && otherValue
	//              ^^^^^^^^^^
}

Synthesizing the right-hand side of this logical expression (otherValue) is done in a new context. This is because it conditionally runs (because of short-circuiting with logical and), any effects here have to be separated and narrowing can because it is known that input has type null here.

The different contexts, `GeneralContext` and `get_on_ctx!`

There are two different types of contexts Root and Environment (aliases for Context<Root> and Context<environment::Syntax<'a>>). Root is the top-level environment, it doesn't have events or a parent. Environments on the other hand represent the standard contexts found for written code. They record events (and other information) and have a parent. You can see the code for all contexts here, and the code specific for environments (not root contexts) here.

This is structured using Rust generics and so can be a bit complex. To make it easier GeneralContext is a sum type of the two contexts. To get properties on it, you can use get_on_ctx!.

Not a huge fan of macro_rules macros, but this seems to be a useful pattern I haven't seen elsewhere!

Looking up information with `parents_iter`

In JS information can be accessed in am above context

const x = 2;
function three() {
	return x + 1;
	//     ^
	// What is the value of `x` here
}
const x = 2;
function three() {
	return x + 1;
	//     ^
	// What is the value of `x` here
}

Context::parents_iter offers a way to walk up the parent chain to look for information.

Where are the properties of types?

One way to have an object type would be to do

struct InterfaceType {
	name: String,
	properties: HashMap<String, TypeId>
	//          ^^^^^^^^^^^^^^^^^^^^^^^
	// Map properties to types
}
struct InterfaceType {
	name: String,
	properties: HashMap<String, TypeId>
	//          ^^^^^^^^^^^^^^^^^^^^^^^
	// Map properties to types
}

This is the way TSC does it

However, the problem is that the value of properties is not always a global fact. In the two scopes

const x = {};
if (value) {
	x.a = 2;
	// here x.a = 2
} else {
	x.a = 3;
	// here x.a = 3
}
const x = {};
if (value) {
	x.a = 2;
	// here x.a = 2
} else {
	x.a = 3;
	// here x.a = 3
}

x.a has different values as shown in the comments.

This is not true for all facts, in the two scopes calling the function x has the same behaviour.

function x() { ... }

if (condition) {
	x()
} else {
	x()
}
function x() { ... }

if (condition) {
	x()
} else {
	x()
}

Calling x (the value not the reference) always yields the same as x (and that can differ).

Similar logic is done for the value variables, which are stored in variable_current_value.

So Ezno these are held under the context. While not implemented, prototypes and type narrowing will also work this way, being scoped to contexts rather than global.

This is done for all types for simplicity, including interfaces.

The variable and property facts applies to many other dynamic and static languages, not just JavaScript.

Scopes and dynamic scopes

As said before, every environment has a scope type. When looking for things between scopes, it needs to know whether the facts are constant between the scopes. For example

let x = 2;
if (condition) {
    let y = x;
    //      ^
    // x is always 2 here
}
let x = 2;
if (condition) {
    let y = x;
    //      ^
    // x is always 2 here
}

However for

let x = 2;
function func() {
     let y = x;
     //      ^
     // x may not be 2 here if it is assigned a different value before a `func` function call
}
let x = 2;
function func() {
     let y = x;
     //      ^
     // x may not be 2 here if it is assigned a different value before a `func` function call
}

and

let x = 2;
for (const item of myArray) {
    let y = x;
    //      ^
    // x can be different again, because of the following statement
    x++
}
let x = 2;
for (const item of myArray) {
    let y = x;
    //      ^
    // x can be different again, because of the following statement
    x++
}

The first example of if is a static scope. Whereas for the function and for scopes they are considered dynamic scopes and so the context logic does different behaviour in many cases when crossing these contexts.

The main global structures: `CheckingData`, `DiagnosticsContainer`, `TypeMappings` and `TypeStore`

CheckingData holds most information for a project it holds DiagnosticsContainer and TypeStore
DiagnosticsContainer is a container of Diagnostics that includes errors, warnings and general information.
TypeStore stores all the types
- Currently, types are referenced using pub struct TypeId(u16) where the integer references an index into a Vec<Type>
- This is a sort of arena. It means that types can safely reference themselves. While it does hold a lot of memory, types are needed between passes, so there isn't a lot to be dropped. It also means that it is easy to be serialized to a format that can be stored between runs to make checking times improved.
- This is a work in progress, it will be improved to split types by their file/source and probably other ways.

`Type` and `TypeId`

One thing is types are immutable, once they are registered the actual Type doesn't change! There are lots of kinds of Types

The basics are:

Named types (interfaces)
Aliases
Or (union types), these mean that they (inclusively) either have the properties on the left or on the right
And (intersection types), these mean it satisfies the properties on the left and the right
Constants/terms
Functions
Objects
Root poly (parameters)
Constructors (usage of parameters)

Categorising types

There are many types that are considered restrictions/general spaces. These cannot be variable or property current values!

Interfaces
Intersection and unions
Functions from type annotation

There are three types that are considered constants and so have a consistent total known behaviour

Type::Constant which represents numbers, strings, and booleans
Functions defined in the source
Immutable objects and objects where access does not cross a dynamic scope.

The bridge between the two are poly-types

Parameter types (poly-types)

Constants are great however, places where something can't be represented as a constant are known as poly-types. Function parameters are one example, also explicit generics.

Parameters

Alongside explicit function parameters (such as x in function func(x) { ... }) there are other places in JS where a function can be parameterized. For example, closed over variables and this. These are the most common poly-types.

Constructors, the higher poly-types

Parameters are root poly-types. All that is known is the origin. However, we want to carry more information. This is where constructors come in, they wrap any poly type with additional information about its usage.

Closed and open poly-types

The standard poly-type is closed. These are types such that a set of specializations can be formed.

let a = 1;
function id<T>(t: T) {
	return { t, a }
}

id("hi");
a++;
id("hello");
let a = 1;
function id<T>(t: T) {
	return { t, a }
}

id("hi");
a++;
id("hello");

After checking the above script, a in the function body is specialized as 1 and 2 and T is specialized as "hi" and "hello". In other words, we know a set of values that poly type takes throughout the life of the script.

The tracing problem

When evaluating let p = document.body, document can take on a range of values and so can document.body. So in this way they are similar to regular function parameters. In fact, if JS had a main function without global variables it might look like:

function main(document, fetch, ...) {
	// ...
}
function main(document, fetch, ...) {
	// ...
}

So variables such as document are considered similar to parameters. These are known as open poly-types. The open refers to the fact that what they are specialized to is out of the range of the checker. After checking, closed poly-types have a finite set of types they are specialized as. On the other hand, open have an unknown set.

This is still experimental and will probably be changed in some way at some point. Some open poly traces are completely useless, such as Math.

Is subtype

The type_is_subtype function compares types. It is used when checking an argument is valid against a parameter, or a value assigned to variable meets it's constraint. It returns the following result:

#[derive(Debug)]
pub enum SubTypeResult {
	IsSubtype,
	IsNotSubType(NonEqualityReason),
}
#[derive(Debug)]
pub enum SubTypeResult {
	IsSubtype,
	IsNotSubType(NonEqualityReason),
}

The function checks whether the RHS has all the 'properties' of the LHS.

Specialisation

In TypeScript (and Rust), call site generics can be inferred. This means that when calling a function with generic type parameters, the type parameters can be derived from the types of arguments if they were not already explicitly given

function identity<T>(a: T) {
	return a
}

const x = identity(console)
function identity<T>(a: T) {
	return a
}

const x = identity(console)

The other thing is that in Ezno, every function parameter (apart from functions behind a generic) is generic (cascade information). So these auto generics cannot be explicit set for parameters not explicitly generic.

Ezno could resolve these types by scanning the argument list first. Instead during equality, while checking it adds these values to a SeedingContext.

Constraint inference

This is something being worked out. Most parameters have a fixed and known space they operate in. However, to enable full checking on sources without annotation, this restriction could be generated from its usage. See more in the issue.

const sinPlusOne = (x) => {
    return Math.sin(x) + 1
    //     ^^^^^^^^^^^
    // This call requires `x` to be a number. The poly
    // constraint needs to be modified to reflect this
}
const sinPlusOne = (x) => {
    return Math.sin(x) + 1
    //     ^^^^^^^^^^^
    // This call requires `x` to be a number. The poly
    // constraint needs to be modified to reflect this
}

Constant functions

Some functions have a constant identifier (as shown above in the return annotation). Functions with this try and evaluate a result using custom Rust code. The Rust code only works for constants and so if it can't compute a result it will fail and resort back to specializing the return type of the function.

This is what enables the following error to be caught

Events and effects

Events track mutations and other actions the code does. There are many kinds of events that can happen, the simple ones are assignments and function calls.

Events are the general term for these. When added to a FunctionType these are referred to as (side) effects of a function. You can see the events being pulled here.

Throw and try-catch

Following on from a previous post, I managed to get try-catch statements to work and be type-safe 🎉

While most of the time Error is thrown, this covers every type. With dependent string types you also get the message of what types are thrown.

The above is a bit more complicated because it is throwing a parameter type. The actual value is found through type specialisation (in the same way that let x: 2 = (a => a)(2) works). The types of errors thrown are found using this function (which again is work-in-progress).

More benefits of the events system

They reuse types so can be specialized like the above
Catch a lot of side effect cases, which can give more information rather than resorting to unknown-ness
Calling functions to track what functions are called, which allows tree shaking of associated functions (functions under objects)
They could be used as IR for generating more optimized output

Some extras

Hoisting passes

Currently work in progress, but you can see how interfaces are checker before functions which is checked before any other structure in hoist_statements

Printing types

Turning types into strings is done here.

This also has to handle some other things:

Recording what types have been printed types to not stack overflow in the case of cyclic types

`Diagnostic`s, registering errors and warnings

diagnostics.rs has a long list of errors and how to turn them into general Diagnostics which can be printed to the CLI or presented in a LSP.

Developing

If you are interested in contributing, read CONTRIBUTING.md.

What is next

You can read more standing issues on GitHub issues.

There is still more to do

Ezno's own CLI, with the CLI REPL
Finishing off the LSP (language service provider) with an associated VS Code extension to get type-checking results and helpers inside an editor.
Using the types in optimising compilers!
Some more experimental features in terms of syntax and things 👀
Finishing Ezno's own parser, it is 95% there, so it would be nice to round it off

However I am now taking a break

Unfortunately, I have other priorities for the next two months, so I can't spearhead additions and fixes. I will be around though to respond to certain things but development will be slower.

Incredibly grateful for current sponsors. Any contributions including one-time are important for this project to keep going. For now, GitHub sponsors is looking like the easiest way to do this.

Hopefully, I can keep this going as a chill experimental project without strings attached that I can keep improving part-time!

My Rust infrastructure

2023-04-20T00:00:00Z

I have written a lot of Rust over the last couple of years. Along the way of building a compiler, I have built up a few smaller, generic crates and tools (infrastructure) to assist with writing Rust. I realised I haven't shared much about some of these, so I thought it would be a good opportunity to give an overview now.

If you are interested in getting started with Rust: Last summer I wrote a collection of posts for Shuttle (a great place to deploy Rust server applications). I wrote a bit about patterns with Rust types, how Rust tackles error handling or the builder pattern as well as many others. If you are looking to get started with Rust, check those posts out as well as their platform (IMO currently the easiest way to deploy a Rust server application).

This post was meant to part of an upcoming post about the parser I just published, but this stuff didn't fit into that post. So I decided to split the content up. Look forward to a future post which includes things I learned about parsing and talks about some parser utility libraries I have written (source-map, tokenizer-lib and derive-finite-automaton) etc.

I'll start off with a couple of libraries I've built that make it easier and shorter to write Rust code.

Building macros with syn-helpers

Proc(edural) macros are a way of generating Rust code. Derive proc macros are Rust's approach to reflection. I wrote at length about reflection, the use cases and an example with a comparison between vanilla JavaScript and Rust in this post on Shuttle.rs. In summary, proc derive macros allow for generating the Rust impl blocks based on the content of struct and enum declarations. For example, #[derive(Debug)] above a struct declaration finds fields and generates an impl Debug for ... with an implementation that prints the field name alongside its runtime value.

One thing about proc macros is that you will likely be reaching for the dependencies syn and quote when writing them. The API for writing them only gives us a low-level sequence of tokens. Fortunately syn exists and can parse the sequence into an AST, which makes it a lot easier to read fields. Generating output code is also made easier as quote offers a declarative using a macro: quote! (parse_quote! is also a similar thing and equally useful).

But just these on their own, it can still be difficult and verbose to build actual proc macros (the code in the post only worked for structs with named fields). Syn and quote offer great building blocks but they don't give you much help handling the setup required for derive proc macros.

Some of the problems I have run that make the code harder and longer to write and can also introduce bugs:

Writing logic that can handle both structs and enums and their variants. You can use self to reference data in a struct, but not in an enum
Creating expressions that access both named fields and unnamed fields
Handling generics that exist on either of the structure or the trait and how to handle if the names clash
How to handle attributes, how to access them at different levels
Forgetting to add #[automatically_derived]

To make this easier, last year I wrote a 'procedural macro framework' that abstracted syn and this process to handle all these cases for you. It's a bit difficult to explain in words so here is an example that calls do_thing for all fields except ones marked with ignore.

use syn_helpers::{
    syn::{parse_quote, DeriveInput, GenericParam, Ident, Stmt}, proc_macro2::Span, quote,
    derive_trait, FieldMut, HasAttributes, Trait, TraitItem, TypeOfSelf, Constructable,
};

let my_trait = Trait {
    name: parse_quote!(::my_crate::MyTrait),
    generic_parameters: None,
    items: vec![TraitItem::new_method(
        Ident::new("method_one", Span::call_site()),
        None,
        TypeOfSelf::Reference,
        Vec::default(),
        None,
        |mut item| {
            item.map_constructable(|mut constructable| {
                Ok(constructable
                    .get_fields_mut()
                    .fields_iterator_mut()
                    .flat_map(|mut field| -> Option<Stmt> {
                        if field
                            .get_attributes()
                            .iter()
                            .any(|attr| attr.path.is_ident("ignore"))
                        {
                            None
                        } else {
                            let reference = field.get_reference();
                            Some(parse_quote!(do_thing(#reference);))
                        }
                    })
                    .collect())
            })
        },
    )],
};

let r#struct: DeriveInput = parse_quote! {
    struct X {
        a: String,
        b: i32
    }
};

let stream = derive_trait(r#struct, my_trait);

assert_eq!(
    stream.to_string(),
    quote! {
        #[automatically_derived]
        impl ::my_crate::MyTrait for X {
            fn method_one(&self) {
                let X { a: ref _0, b: ref _1 } = self;
                do_thing(_0);
                do_thing(_1);
            }
        }
    }.to_string()
)
use syn_helpers::{
    syn::{parse_quote, DeriveInput, GenericParam, Ident, Stmt}, proc_macro2::Span, quote,
    derive_trait, FieldMut, HasAttributes, Trait, TraitItem, TypeOfSelf, Constructable,
};

let my_trait = Trait {
    name: parse_quote!(::my_crate::MyTrait),
    generic_parameters: None,
    items: vec![TraitItem::new_method(
        Ident::new("method_one", Span::call_site()),
        None,
        TypeOfSelf::Reference,
        Vec::default(),
        None,
        |mut item| {
            item.map_constructable(|mut constructable| {
                Ok(constructable
                    .get_fields_mut()
                    .fields_iterator_mut()
                    .flat_map(|mut field| -> Option<Stmt> {
                        if field
                            .get_attributes()
                            .iter()
                            .any(|attr| attr.path.is_ident("ignore"))
                        {
                            None
                        } else {
                            let reference = field.get_reference();
                            Some(parse_quote!(do_thing(#reference);))
                        }
                    })
                    .collect())
            })
        },
    )],
};

let r#struct: DeriveInput = parse_quote! {
    struct X {
        a: String,
        b: i32
    }
};

let stream = derive_trait(r#struct, my_trait);

assert_eq!(
    stream.to_string(),
    quote! {
        #[automatically_derived]
        impl ::my_crate::MyTrait for X {
            fn method_one(&self) {
                let X { a: ref _0, b: ref _1 } = self;
                do_thing(_0);
                do_thing(_1);
            }
        }
    }.to_string()
)

Benefits of using the crate here

map_constructable handles both enums variants and struct-ures.
The code uses the Field trait which abstracts over generating expressions to access fields. Handling named and unnamed variants is automated away
For fields whose type contains generics, it can add necessary where clauses if the trait is called on that field
The macro also gets a structure. You get a more declarative code by laying out the Trait with the items you need to implement. Those methods contain functions that handle generating the output.

For example, in the parser, I currently have a way of visiting nodes (running a set of functions over them) and automating this implementation becomes quite simple using the syn-helpers library.

use proc_macro::TokenStream;
use std::error::Error;
use string_cases::StringCasesExt;
use syn_helpers::{
    derive_trait,
    proc_macro2::{Ident, Span},
    quote::{self, format_ident},
    syn::{parse_macro_input, parse_quote, DeriveInput, Stmt,
    Constructable, FieldMut, HasAttributes, NamedOrUnnamedFieldMut,
		Trait, TraitItem,
};

/// On the top structure
const VISIT_SELF_NAME: &str = "visit_self";
/// Per field modifiers
const VISIT_SKIP_NAME: &str = "visit_skip_field";
/// Add to chain. Can be on item or a field
const VISIT_WITH_CHAIN_NAME: &str = "visit_with_chain";

/// Usage #[derive(Visitable)]
#[proc_macro_derive(
    Visitable,
    attributes(visit_self, visit_skip_field, visit_custom_visit)
)]
pub fn generate_visit_implementation(input: TokenStream) -> TokenStream {
    let input = parse_macro_input!(input as DeriveInput);

    let visit_item = TraitItem::new_method(
        Ident::new("visit", Span::call_site()),
        Some(vec![parse_quote!(TData)]),
        syn_helpers::TypeOfSelf::Reference,
        vec![
            parse_quote!(visitors: &mut (impl crate::visiting::VisitorReceiver<TData> + ?Sized)),
            parse_quote!(data: &mut TData),
            parse_quote!(settings: &crate::VisitSettings),
            parse_quote!(chain: &mut ::temporary_annex::Annex<crate::visiting::Chain>),
        ],
        None,
        |item| generated_visit_item(item, VisitType::Immutable),
    );

    let visit_mut_item = TraitItem::new_method(
        Ident::new("visit_mut", Span::call_site()),
        Some(vec![parse_quote!(TData)]),
        syn_helpers::TypeOfSelf::MutableReference,
        vec![
            parse_quote!(visitors: &mut (impl crate::visiting::VisitorMutReceiver<TData> + ?Sized)),
            parse_quote!(data: &mut TData),
            parse_quote!(settings: &crate::VisitSettings),
            parse_quote!(chain: &mut ::temporary_annex::Annex<crate::visiting::Chain>),
        ],
        None,
        |item| generated_visit_item(item, VisitType::Mutable),
    );

    let visitable_trait = Trait {
        name: parse_quote!(crate::visiting::Visitable),
        generic_parameters: None,
        items: vec![visit_item, visit_mut_item],
    };

    let output = derive_trait(input, visitable_trait);

    output.into()
}

#[derive(Clone, Copy)]
enum VisitType {
    Immutable,
    Mutable,
}

fn generated_visit_item(
    mut item: syn_helpers::Item,
    visit_type: VisitType,
) -> Result<Vec<Stmt>, Box<dyn Error>> {
    let attributes = item.structure.get_attributes();

    let visit_self = attributes
        .iter()
        .any(|attr| attr.path.is_ident(VISIT_SELF_NAME));

    let visit_with_chain = attributes.iter().find_map(|attr| {
        attr.path
            .is_ident(VISIT_WITH_CHAIN_NAME)
            .then_some(&attr.tokens)
    });

    let mut lines = Vec::new();

    if let Some(expr_tokens) = visit_with_chain {
        lines.push(parse_quote!( let mut chain = &mut chain.push_annex(#expr_tokens); ))
    }

    if visit_self {
        let struct_name_as_snake_case = &item.structure.get_name().to_string().to_snake_case();
        let mut_postfix = matches!(visit_type, VisitType::Mutable)
            .then_some("_mut")
            .unwrap_or_default();
        let func_name = format_ident!("visit_{}{}", struct_name_as_snake_case, mut_postfix);

        lines.push(parse_quote!( visitors.#func_name(self, data,  chain); ))
    }

    let mut field_lines = item.map_constructable(|mut constructable| {
        Ok(constructable
			.get_fields_mut()
			.fields_iterator_mut()
			.flat_map(|mut field: NamedOrUnnamedFieldMut| -> Option<Stmt> {
				let attributes = field.get_attributes();

				let skip_field = attributes.iter().any(|attr| attr.path.is_ident(VISIT_SKIP_NAME));

				let visit_with_chain = attributes.iter().find_map(|attr| {
					attr.path.is_ident(VISIT_WITH_CHAIN_NAME).then_some(&attr.tokens)
				});

				let chain = if let Some(expr_tokens) = visit_with_chain {
					quote!(&mut chain.push_annex(#expr_tokens))
				} else {
					quote!(chain)
				};

				if !skip_field {
					let reference = field.get_reference();
					Some(match visit_type {
						VisitType::Immutable => parse_quote! {
							crate::Visitable::visit(#reference, visitors, data, settings, #chain);
						},
						VisitType::Mutable => parse_quote! {
							crate::Visitable::visit_mut(#reference, visitors, data, settings, #chain);
						},
					})
				} else {
					None
				}
			})
			.collect::<Vec<_>>())
    })?;

    lines.append(&mut field_lines);

    Ok(lines)
}
use proc_macro::TokenStream;
use std::error::Error;
use string_cases::StringCasesExt;
use syn_helpers::{
    derive_trait,
    proc_macro2::{Ident, Span},
    quote::{self, format_ident},
    syn::{parse_macro_input, parse_quote, DeriveInput, Stmt,
    Constructable, FieldMut, HasAttributes, NamedOrUnnamedFieldMut,
		Trait, TraitItem,
};

/// On the top structure
const VISIT_SELF_NAME: &str = "visit_self";
/// Per field modifiers
const VISIT_SKIP_NAME: &str = "visit_skip_field";
/// Add to chain. Can be on item or a field
const VISIT_WITH_CHAIN_NAME: &str = "visit_with_chain";

/// Usage #[derive(Visitable)]
#[proc_macro_derive(
    Visitable,
    attributes(visit_self, visit_skip_field, visit_custom_visit)
)]
pub fn generate_visit_implementation(input: TokenStream) -> TokenStream {
    let input = parse_macro_input!(input as DeriveInput);

    let visit_item = TraitItem::new_method(
        Ident::new("visit", Span::call_site()),
        Some(vec![parse_quote!(TData)]),
        syn_helpers::TypeOfSelf::Reference,
        vec![
            parse_quote!(visitors: &mut (impl crate::visiting::VisitorReceiver<TData> + ?Sized)),
            parse_quote!(data: &mut TData),
            parse_quote!(settings: &crate::VisitSettings),
            parse_quote!(chain: &mut ::temporary_annex::Annex<crate::visiting::Chain>),
        ],
        None,
        |item| generated_visit_item(item, VisitType::Immutable),
    );

    let visit_mut_item = TraitItem::new_method(
        Ident::new("visit_mut", Span::call_site()),
        Some(vec![parse_quote!(TData)]),
        syn_helpers::TypeOfSelf::MutableReference,
        vec![
            parse_quote!(visitors: &mut (impl crate::visiting::VisitorMutReceiver<TData> + ?Sized)),
            parse_quote!(data: &mut TData),
            parse_quote!(settings: &crate::VisitSettings),
            parse_quote!(chain: &mut ::temporary_annex::Annex<crate::visiting::Chain>),
        ],
        None,
        |item| generated_visit_item(item, VisitType::Mutable),
    );

    let visitable_trait = Trait {
        name: parse_quote!(crate::visiting::Visitable),
        generic_parameters: None,
        items: vec![visit_item, visit_mut_item],
    };

    let output = derive_trait(input, visitable_trait);

    output.into()
}

#[derive(Clone, Copy)]
enum VisitType {
    Immutable,
    Mutable,
}

fn generated_visit_item(
    mut item: syn_helpers::Item,
    visit_type: VisitType,
) -> Result<Vec<Stmt>, Box<dyn Error>> {
    let attributes = item.structure.get_attributes();

    let visit_self = attributes
        .iter()
        .any(|attr| attr.path.is_ident(VISIT_SELF_NAME));

    let visit_with_chain = attributes.iter().find_map(|attr| {
        attr.path
            .is_ident(VISIT_WITH_CHAIN_NAME)
            .then_some(&attr.tokens)
    });

    let mut lines = Vec::new();

    if let Some(expr_tokens) = visit_with_chain {
        lines.push(parse_quote!( let mut chain = &mut chain.push_annex(#expr_tokens); ))
    }

    if visit_self {
        let struct_name_as_snake_case = &item.structure.get_name().to_string().to_snake_case();
        let mut_postfix = matches!(visit_type, VisitType::Mutable)
            .then_some("_mut")
            .unwrap_or_default();
        let func_name = format_ident!("visit_{}{}", struct_name_as_snake_case, mut_postfix);

        lines.push(parse_quote!( visitors.#func_name(self, data,  chain); ))
    }

    let mut field_lines = item.map_constructable(|mut constructable| {
        Ok(constructable
			.get_fields_mut()
			.fields_iterator_mut()
			.flat_map(|mut field: NamedOrUnnamedFieldMut| -> Option<Stmt> {
				let attributes = field.get_attributes();

				let skip_field = attributes.iter().any(|attr| attr.path.is_ident(VISIT_SKIP_NAME));

				let visit_with_chain = attributes.iter().find_map(|attr| {
					attr.path.is_ident(VISIT_WITH_CHAIN_NAME).then_some(&attr.tokens)
				});

				let chain = if let Some(expr_tokens) = visit_with_chain {
					quote!(&mut chain.push_annex(#expr_tokens))
				} else {
					quote!(chain)
				};

				if !skip_field {
					let reference = field.get_reference();
					Some(match visit_type {
						VisitType::Immutable => parse_quote! {
							crate::Visitable::visit(#reference, visitors, data, settings, #chain);
						},
						VisitType::Mutable => parse_quote! {
							crate::Visitable::visit_mut(#reference, visitors, data, settings, #chain);
						},
					})
				} else {
					None
				}
			})
			.collect::<Vec<_>>())
    })?;

    lines.append(&mut field_lines);

    Ok(lines)
}

It takes around 140 lines to implement this trait. syn-helpers here can handle the cases when the AST contains generics. Using syn-helpers means the code can focus on the actual behaviour of the trait rather than handling all the different cases Rust declarations can be in.

Although it is called syn-helpers and that was its original aim, it has now grown into a large framework focused on derive macros. So if you write derive macros and maybe struggle with keeping things concise. If you have API/abstraction you want to be added for a proc-macro you are writing I am open for discussion.

Building this out I feel it might be close to a fully declarative macro implementation notes here. Rust current has declarative macros but anything derive based has to be done imperatively and there isn't an end-to-end declarative approach. Maybe it could be tried out in this library.

Putting the framework to build more customisable `derive` implementations

I had written two macros before, and syn-helpers helped tidy up and share the code between these two. The two main ones are:

`#[derive(DebugExtras)]`

#[derive(Debug)] is great most times, but sometimes it lacks a bit of customisation for the derive item. derive-debug-extras offers an enhanced macro that adds #[debug_ignore] and #[debug_as_display] attributes that can help improve debugging.

But arguably my favourite one is #[debug_single_tuple_inline]. This fixes a problem in the Ezno checker. I use a lot of the new type pattern to declare identifiers. Unfortunately, despite them only having one field, when debugging them with the pretty flag (I want other named structs to be over lines) it ends up over three. So although it did require rewriting the standard debug implementation, now:

// Without #[debug_single_tuple_inline]
[
    A(
        123
    ),
    A(
        145
    ),
    A(
        125
    ),
]
// With #[debug_single_tuple_inline]
[
    A(123),
    A(145),
    A(125),
]
// Without #[debug_single_tuple_inline]
[
    A(
        123
    ),
    A(
        145
    ),
    A(
        125
    ),
]
// With #[debug_single_tuple_inline]
[
    A(123),
    A(145),
    A(125),
]

IMO, all unnamed fields when debugged should be on one line despite the pretty flag.

`#[derive(PartialEqExtras)]`

In a similar vein, derive-partial-eq-extras adds a more customisable PartialEq implementation. It adds two new attributes for ignoring certain fields in the implementer.

For example, in my parser, expressions have positions and IDs. However, when comparing two expressions I want to treat them on a value basis and not based on position or identifiers. For example, given the literal expression 5, I want AST to equal other 5s, no matter where they have been parsed. To do this I simply add #[partial_eq_ignore_types], which I can use easily on the expression AST:

#[derive(PartialEqExtras, Debug, Clone)]
#[partial_eq_ignore_types(Span, ExpressionId)]
pub enum Expression {
	NumberLiteral(NumberStructure, Span, ExpressionId),
	StringLiteral(String, #[partial_eq_ignore] Quoted, Span, ExpressionId),
	BooleanLiteral(bool, Span, ExpressionId),
	RegexLiteral {
		pattern: String,
		flags: Option<String>,
		position: Span,
		id: ExpressionId,
	},
	ArrayLiteral(Vec<SpreadExpression>, Span, ExpressionId),
	ObjectLiteral(ObjectLiteral),
	...
#[derive(PartialEqExtras, Debug, Clone)]
#[partial_eq_ignore_types(Span, ExpressionId)]
pub enum Expression {
	NumberLiteral(NumberStructure, Span, ExpressionId),
	StringLiteral(String, #[partial_eq_ignore] Quoted, Span, ExpressionId),
	BooleanLiteral(bool, Span, ExpressionId),
	RegexLiteral {
		pattern: String,
		flags: Option<String>,
		position: Span,
		id: ExpressionId,
	},
	ArrayLiteral(Vec<SpreadExpression>, Span, ExpressionId),
	ObjectLiteral(ObjectLiteral),
	...

This is similar to educe, however it does not have the ignore types feature, so declarations can get clobbered with annotations on lots of fields. I did try and add it to educe but got a bit scared by its codebase.

Again look into the sources and see that they are slim as they are built of syn-helpers

Assisting with sum `enum`s

In the parser, I often have types that sum together some struct definitions like

#[derive(Debug, Clone, PartialEq)]
pub enum Declaration {
	Variable(VariableDeclaration),
	Function(Decorated<StatementFunction>),
	Class(Decorated<ClassDeclaration<StatementPosition>>),
	Enum(Decorated<EnumDeclaration>),
	...
}
#[derive(Debug, Clone, PartialEq)]
pub enum Declaration {
	Variable(VariableDeclaration),
	Function(Decorated<StatementFunction>),
	Class(Decorated<ClassDeclaration<StatementPosition>>),
	Enum(Decorated<EnumDeclaration>),
	...
}

Converting in and out can be a bit of a pain. If I have a function that takes a Declaration I need to remember the variant and then wrap the whole expression in it. So I created derive enum from into. With this, I can add #[derive(EnumFrom, EnumTryInto)] to have an easier time converting between the two (including immutable and mutable references).

#[derive(Debug, Clone, EnumFrom, EnumTryInto, PartialEq)]
#[try_into_references(&, &mut)]
pub enum Declaration {
	Variable(VariableDeclaration),
	Function(Decorated<StatementFunction>),
	Class(Decorated<ClassDeclaration<StatementPosition>>),
	Enum(Decorated<EnumDeclaration>),
	...
}

fn my_func(var_dec: VariableDeclaration) {
	let dec = Declaration::from(var_dec);
	assert!(matches!(dec, Declaration::Variable(_)));
	let result = Decorated::<StatementFunction>::try_from(dec);
	assert!(result.is_err());
}
#[derive(Debug, Clone, EnumFrom, EnumTryInto, PartialEq)]
#[try_into_references(&, &mut)]
pub enum Declaration {
	Variable(VariableDeclaration),
	Function(Decorated<StatementFunction>),
	Class(Decorated<ClassDeclaration<StatementPosition>>),
	Enum(Decorated<EnumDeclaration>),
	...
}

fn my_func(var_dec: VariableDeclaration) {
	let dec = Declaration::from(var_dec);
	assert!(matches!(dec, Declaration::Variable(_)));
	let result = Decorated::<StatementFunction>::try_from(dec);
	assert!(result.is_err());
}

This is a smaller, less configurable, enum only version of #[derive(From)] from the derive_more crate

This crate is actually my most downloaded, possibly because it seems to be used on this popular project.

Iterator endiate

A smaller one, but I often have a case when iterating through something I want to know if it is the last one. This often happens in the to-string part of my parser where I want to add a comma delimiter between items but don't want a trailing comma. For iterators with a known size, this crate adds an extension trait that adds the endiate method. Similar in functionality to the enumerate.

// adds `.endiate()` method to all (sized) iterators
use iterator_endiate::EndiateIteratorExt;

for (at_end, item) in items.iter().endiate() {
	settings.add_indent(depth, buf);
	item.to_string_from_buffer(buf, settings, depth);
	if !at_end {
		if item.requires_semi_colon() {
			buf.push(';');
		}
		if settings.pretty {
			buf.push_new_line();
		}
	}
}
// adds `.endiate()` method to all (sized) iterators
use iterator_endiate::EndiateIteratorExt;

for (at_end, item) in items.iter().endiate() {
	settings.add_indent(depth, buf);
	item.to_string_from_buffer(buf, settings, depth);
	if !at_end {
		if item.requires_semi_colon() {
			buf.push(';');
		}
		if settings.pretty {
			buf.push_new_line();
		}
	}
}

Also adds the nendiate for when you want to know you are not at the end

Enums and strings

Enum variants strings is a library for converting between (yes, both ways) &str and enum structures. The derive macro handles generating this based on variant names. Mapping from a string to an enum, works for simple things that implement Default.

use enum_variants_strings::EnumVariantsStrings;

#[derive(Debug, PartialEq, EnumVariantsStrings)]
enum Variants {
    X,
    Y(i32),
    #[enum_variants_strings_mappings("z", "zee")]
    Z {
        x: String,
        y: String,
    },
}

fn main() {
    assert_eq!(Variants::from_str("x"), Ok(Variants::X));
    assert_eq!(Variants::from_str("y"), Ok(Variants::Y(0)));
    assert_eq!(
        Variants::from_str("z"),
        Ok(Variants::Z {
            x: String::default(),
            y: String::default(),
        })
    );

    assert_eq!(Variants::X.to_str(), "x");
    assert_eq!(
        Variants::Z {
            x: "abc".into(),
            y: "xyz".into()
        }
        .to_str(),
        "zee"
    );
}
use enum_variants_strings::EnumVariantsStrings;

#[derive(Debug, PartialEq, EnumVariantsStrings)]
enum Variants {
    X,
    Y(i32),
    #[enum_variants_strings_mappings("z", "zee")]
    Z {
        x: String,
        y: String,
    },
}

fn main() {
    assert_eq!(Variants::from_str("x"), Ok(Variants::X));
    assert_eq!(Variants::from_str("y"), Ok(Variants::Y(0)));
    assert_eq!(
        Variants::from_str("z"),
        Ok(Variants::Z {
            x: String::default(),
            y: String::default(),
        })
    );

    assert_eq!(Variants::X.to_str(), "x");
    assert_eq!(
        Variants::Z {
            x: "abc".into(),
            y: "xyz".into()
        }
        .to_str(),
        "zee"
    );
}

I use it for changing between modes in the ast-playground of Ezno.

Deploying Rust with GitHub actions

In my crates, I want the update commit to automatically push the version change back to the repository. If I do it manually, I always forget to push after, things get out of sync.

So I built a GitHub action for doing this: crates-release-action

Here is how you can use it:

name: Release crate

on:
  workflow_dispatch:
    inputs:
      version:
        description: "major/minor/patch or semver"
        required: false
        default: "patch"

concurrency: release-crate

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set git credentials
        run: |
          git config user.name github-actions
          git config user.email github-actions@github.com
      - name: Crates publish
        uses: kaleidawave/crates-release-gh-action@main
        id: release
        with:
          version: $
          crates-token: $
      - name: Push updated Cargo.toml
        run: |
          git add .
          git commit -m "Release: $"
          git tag "release/$"
          git push --tags origin main
name: Release crate

on:
  workflow_dispatch:
    inputs:
      version:
        description: "major/minor/patch or semver"
        required: false
        default: "patch"

concurrency: release-crate

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set git credentials
        run: |
          git config user.name github-actions
          git config user.email github-actions@github.com
      - name: Crates publish
        uses: kaleidawave/crates-release-gh-action@main
        id: release
        with:
          version: $
          crates-token: $
      - name: Push updated Cargo.toml
        run: |
          git add .
          git commit -m "Release: $"
          git tag "release/$"
          git push --tags origin main

It handles:

Finding manifests
Applying major, minor or patch to a version (or an exact version) and updating the contents of Cargo.toml
Finding local manifests that reference it as a path dependency to update their version
Publishing to crates.io
Outputting the new version(s) in a machine-readable format (so it can be referenced in commit names and tags)

You can run it through github.com

Or from the command line with gh

Which I can watch

Behind the scenes, it updates the packages in the order of least dependency (I don't want to rely on myself for ordering arguments, it can be calculated). It also uses a TOML parser that retains the TOML formatting. It is currently written in Python. If anyone wants to rewrite it in Rust and/or add more functionality, LMK!

It is a manual action currently, and isn't particularly automated. It doesn't track changes in a workspace, or figure out Semver or build changelogs. I want the base to be simple and un-opinionated

I didn't have space but two more crates I have made are: multiline-term-input, which is a way to break into new lines during console input (I need a wiz to add Linux support) and temporary-annex that helps to work with appending data temporarily while reusing the same backing (linear) buffer.

And that is all. If you have built or used any cool Rust libraries or additional tools let me know in the comments!

Ezno in '23

2023-01-18T00:00:00Z

It's been a minute since the previous announcement so I thought would give some updates and share some upcoming problems. This follows the initial announcement and includes some smaller things I shared on Twitter since the announcement post.

Never heard of Ezno? It is a parser, partial executor, optimizer and type checker for JavaScript! Read the initial announcement.

New changes

Classes, getters and setters

I added support for classes, getters and setters

This required some changes to the definition of types and how they handle prototypes. I also added handling for the cases when a property is a getter or setter when doing property access and assignments. This addition also added support for getters and setters in object literals.

Tree shaking methods

When adding classes I extended the tree-shaking mechanism (present but not mentioned in the announcement post) to class methods. Because of object tracing and call detection, if a method on a class is never called then the code is never included in the output.

This is one of the reasons for not following TypeScript's rules around any. If even a single call like (x as any)["method" as any]() was treated with type annotations being the source of truth then any sort of method removal could remove called code.

The perhaps underrated benefit here it is not just class code being removed, it is all functions that might be referenced in a class method that can now be removed.

Custom element registration event

Putting aside any of my opinions on custom elements. Calling CustomElementRegistry.define() results in running an effect that adds a mapping to the HTMLElementTagNameMap type and thus can get type safety in some more tricky scenarios.

A REPL

A REPL (read evaluate print loop) is an interactive code executor. Similar to ts-node, the Ezno REPL wraps an existing REPL with a checker. This required a few changes and while still a long way from perfect the checker can do a little more incrementally.

There is also a mode that prints the type output.

This is mostly to help me with debugging what Ezno does. But maybe it will be useful generally for working out the arguments for functions etc.

You can see the wrapper code for Deno here.

The `satisfies` operator

There are two interesting things I slipped into the first code snippet that ran in the REPL.

The first was a divides operator. I was just playing around, it won't be in the published tool. 😀

The second one is revealed if you try running the code in TSC.

I was recently asked whether Ezno supported the recent TypeScript addition, the satisfies operator. The satisfies binary operator is a compile-time assertion that the left expression operand is typeable under the right type reference operand. It has almost identical behaviour to the assertType identity function trick that was shown in the original announcement.

The existence of this piece of syntax seems to be handling a problem that occurs with variable declarations in TypeScript. With the statement let x: A = b, variable reference x is considered to have all the properties of type A. While that is fine and how every nominal type system works, it can cause problems when A is a larger type than the type of b and loses some properties.

In the code shown in the REPL, TS treats x.prop as string | number. However looking at the code it is clear that the x.prop is a number at the point of the Math.pow call, never a string. What satisfies allows for is checking without widening the type (as widening loses information).

So how does this check under Ezno? Although Math.pow can take a string argument in JS, it is typed as (a: number, b: number) => number;. So it has approved x.prop as a number.

In Ezno variables have two pieces of type data attached to them:

A current value
A space the value can operate in (defined using the variable type annotation)

The current value type is used when checking usage and the space type is used when checking reassignments.

Therefore the Math.pow call checks x using the current value (which is a number here). It does not use the reassignment constraint.

This was present early in Ezno (before I had heard of the satisfies operator). This behaviour was needed to be able to detect dead code here:

let x: number = 2;
if (x !== 2) {
	bigFunction()
}
let x: number = 2;
if (x !== 2) {
	bigFunction()
}

The problem I have with satisfies is that isn't a drop in replacement. There doesn't seem to be a way to retain the value information while constraining variable value for reassignment. That is something that is the default with Ezno's point-in-a-space system.

I am glad that TypeScript is pursuing more into dependent/literal types checking. And I can see why variable declarations are checked that way in TSC. Ezno's implementation only works because of effect tracking, which allows it to identify variable mutations from functions. Effect tracking is tricky to do (as will be shown later), so I understand why TypeScript doesn't do it.

So satisfies doesn't have any effect on checking in Ezno. Similarly things like as const don't help Ezno. They don't have any effect on the system as it is all inferred or treated computationally.

Is there a limit to how much TypeScript can tighten up by adding additional syntax? Does Ezno's effect tracking make it more approachable for beginners?

Binary context/definition files

I added a binary serialized form of contexts just before the announcement went up but didn't have a lot of space to include it. These are an alternative form to .d.ts files and are more compact and include direct references to the identifiers used in Ezno.

In TSC cold start time is not great. One of the bottlenecks (I think) for performance, is the parsing and synthesizing initial contexts stage. Definition files are large, contain lots of unused information in terms of comments (only useful when using TSC in LSP mode) and are a format that is intensive to parse. On the other hand, Ezno's additional low-level byte definition files (that can be generated from an existing context) can be synthesized without having to build up an abstract syntax tree and are much smaller with no whitespace and extracted comments. The format is split up the same way the data structures are arranged in the checker. Regular definition files are dense with information, the binary form has sections for type name mapping, variable mapping, function types, etc which could be processed in parallel for further improvement.

These results are only for simple.d.ts which is a subset of the declarations in lib.d.ts. It's about 6x times smaller as it only includes the definitions that Ezno currently understands. However, the benefits are looking promising at around ~14x faster creating initial contexts:

Parsing .d.ts: 4.4306ms
Synthesizing .d.ts: 10.4137ms

Reading from binary 988.5μs
Parsing .d.ts: 4.4306ms
Synthesizing .d.ts: 10.4137ms

Reading from binary 988.5μs

One of the benefits of being written in Rust is that dealing with low-level byte processing and access is nice and simple!

Still unsure how the distribution of these will work. Also currently it only supports root contexts, haven't figured source splitting and referencing for child contexts yet.

Upcoming changes

Here I some things I am going to start tackling over the next couple of months. They all relate to effects.

Internal object effects

One of the upcoming problems to solve is arrays and mutations.

const x = [1, 2, 3];
x.push(4);
if (x.at(-1) !== 4) {
	console.log("Hi")
}
const x = [1, 2, 3];
x.push(4);
if (x.at(-1) !== 4) {
	console.log("Hi")
}

Here .push mutates x. Just looking at the TS definition for the method, push(...items: T[]): number, it doesn't give much information about what it does. Without knowing what .push mutates the x.get(-1) call can't return accurate information. If it did know, the checker could find 4 !== 4 is always false and the if branch never fires.

Therefore the type declaration needs to include more information. Currently, this is done using decorators on the method declaration, which Ezno interprets specially. However, there may be a more descriptive way to register this information:

interface Array<T> {
	@Effects(
		"sets [this.length] on this to T",
		"sets 'length' on this to Add<this['length'], 1>"
	)
	push(t: T): Add<this["length"], 1>;

	// Maybe it is more readable to do something like
	push(t: T) performs {
		this[this.length] = t;
		return ++this.length;
	}
}
interface Array<T> {
	@Effects(
		"sets [this.length] on this to T",
		"sets 'length' on this to Add<this['length'], 1>"
	)
	push(t: T): Add<this["length"], 1>;

	// Maybe it is more readable to do something like
	push(t: T) performs {
		this[this.length] = t;
		return ++this.length;
	}
}

This shouldn't be impossible to add to declarations. There isn't a huge amount in the standard library and not a huge amount in DOM API. It isn't a hugely complex task for these to be added but is very beneficial.

Asynchronous effects

There are several different types of effects. One is modifying a variable (or property), this fixes several problems with issues with control flow analysis in TypeScript.

So far synchronous situations are covered:

let x = 2;
(() => x++)();
assertType<3>(x);
let x = 2;
(() => x++)();
assertType<3>(x);

Asynchronous code (where the order of execution isn't trivial) is next to implement:

let x = 3;
setTimeout(() => { x++; }, 0);
// Function will be called but not until it has reached a point where the event loop runs
assertType<3>(x)
let x = 3;
setTimeout(() => { x++; }, 0);
// Function will be called but not until it has reached a point where the event loop runs
assertType<3>(x)

To support this, it shouldn't be too difficult:

Determining when functions are queued or registered to an event loop (such as setTimeout, addEventListener) vs immediately called (such as in Array.prototype.map)
Running function effects at the end of synchronisation. Whether that is the end of the script or some await expressions.

Handling errors

In TS, functions that throw can be typed using a return type of never. However, there is no information in the type system to depict what type is thrown. In Ezno throw statements current queue special throw effects. There are two things left before this system becomes usable:

Expressions that always throw and are not in a catch-able structure should produce an error
In a try-catch statement, the checker needs to collect possible objects thrown to form a type of err in catch (err). This makes catch branches TypeSafe™. Or alternatively, if no throw effects in the try block the checker should emit a warning.

And in code that is generic:

function safeEval<T extends () => void>(t: T) {
	try {
		t()
	} catch (err) {
		console.log(err.message)
	}
}
function safeEval<T extends () => void>(t: T) {
	try {
		t()
	} catch (err) {
		console.log(err.message)
	}
}

There needs to exist a Thrown<T> internal helper type that can extract what is thrown from a function.

Is it open source? Where's the binary?

UPDATE, Ezno is now open-source

Open source is more than switching a GitHub repo from private to public. IMO large public repositories should have some sort of stable roadmap or design that contributors can follow. Time needs to be put in to follow issues through. And currently, for the checker, I don't have any of that soo. I think it will be best if it is incrementally made public. The parser is nearly at a point I would consider publishable so hopefully there will be a release soon!

In terms of the executable binary, I want to release it when it can check an actual useful program. Once the above issues with effects are solved then, it will open up more real-world demos. After then, it should be good to go!

Other things coming up

I work on a few other crates. Some of those will be getting updates soon.
I have two posts coming out soon about Rust and procedural macros.

Introducing Ezno

2022-09-22T00:00:00Z

Ezno is an experimental compiler I have been working on and off for a while. In short, it is a JavaScript compiler featuring checking, correctness and performance for building full-stack (rendering on the client and server) websites.

This post is just an overview of some of the features I have been working on which I think are quite cool as well an overview on the project philosophy ;)

It is still work in progress, all the examples work but the tool is still in its infancy

Type synthesis and checking

The core of Ezno is a type checker for JavaScript. Type synthesis is analysing syntax and formulating properties of terms. Comparing the information on terms and how those terms are used, a type checker can prevent errors at runtime.

Ezno's type checker is built from scratch. Getting the features I wanted required a lot of different functionality and needed several new ideas that as far as I know aren't present in any system or existing checker. The checker is fully compatible with TypeScript type annotations and can work without any type annotations at all.

You can think of it as an extension to TSC, similar ideas but taken further

The next few sections go into some unique features of the checker before going into the real benefit of having all this information about types.

Dependent typing

One of the key ideas with Ezno is that it attempts "maximum" knowledge of a source. This knowledge includes:

Runtimes exception that could happen because of missing properties
Code that will never run
Expressions that could be collapsed to reduce work
Mutations to data

Because of the dynamism of JavaScript, this requires including references to constants (aka known numbers, booleans, strings, etc) in the type system.

While TypeScript includes dependent types, Ezno has built on top of that doing more such as constant operator evaluation.

const x: 5 = 4 + 2;
const x: 5 = 4 + 2;

Scaling this up you can now see this working for both identifying an invalid property and detecting dead code:

const obj = {
	key3() { return { someProp: 2 } }
}

obj["key" + (2 + 1)]().nonProp;

const five = 5;
if (five + 5 !== 10) {
	// do stuff
}
const obj = {
	key3() { return { someProp: 2 } }
}

obj["key" + (2 + 1)]().nonProp;

const five = 5;
if (five + 5 !== 10) {
	// do stuff
}

Objects

As well as identifying terms like 4 and "hello", Ezno also treats objects and functions as constant terms:

if ({} === {}) {
	// dead code, two different objects
}

const a = {};
const b = a;
if (a === b) {
	// always true, reference to same object
}

function func() {}
function func2() { return func; }

if (func === func2()) {
	// always true 👀
}
if ({} === {}) {
	// dead code, two different objects
}

const a = {};
const b = a;
if (a === b) {
	// always true, reference to same object
}

function func() {}
function func2() { return func; }

if (func === func2()) {
	// always true 👀
}

Making all function parameters generic/dependent

While the examples showcase great static analysis of a sequence of statements. Synthesis can be more difficult when it breaks into functions and knowing what happens across call sites. Ezno can trace the flow of data and actions on it by treating every parameter as what most languages refer to as generic:

function addOne(x: number): number {
	return x + 1;
}

const three = 3;
const four = addOne(3);

assertType<4>(four);
function addOne(x: number): number {
	return x + 1;
}

const three = 3;
const four = addOne(3);

assertType<4>(four);

Here the addOne function is annotated that it returns a number. But instead internally in Ezno the actual result is what it synthesised that it returned. The number in the return type is only used as a constraint on the type returned in body. The synthesised return type that Ezno uses in this case is Add<T, 1>. In total Ezno internally views the function as: addOne: <T extends number>(x: T): Add<T, 1>

Untyped parameters

Parameters that don't have a type annotation have inferred constraints based on usage in the function body. The simplest is a function with no constraints:

function id(a) {
	return a
}

assertType<number>(id(2));
assertType<string>(id("Hello World"));
assertType<"x">(id("x"));
function id(a) {
	return a
}

assertType<number>(id(2));
assertType<string>(id("Hello World"));
assertType<"x">(id("x"));

Here when the id function is synthesized it infers the a parameter as being generic and thus the function takes the type <T extends any>(a: T) => T.

Inferred generic restrictions

In the above usage, the constraint of parameter a is initialized as any. Its usage in the block didn't require narrowing it down so it stayed as any. Now moving on to a slightly more complex function:

function runMap(obj, func) {
	return obj.map(func)
}
function runMap(obj, func) {
	return obj.map(func)
}

Here initially obj and func are generics that alias any. However, usage of the obj parameter has inferred for the runMap to be type safe obj must have a map property and that map property must be callable with func:

runMap({}, v => v);
runMap({}, v => v);

This is because the .map synthesis changed the alias. The function is synthesized to <T extends { map: (a: U) => any }, U>(obj: T, func: U) => T["map"](U). (where the return type is the call of obj.map with parameter U).

assertType<Array<string>>(runMap(myArray, v => v));
assertType<4>(runMap(
	{ map(cb) { return cb(2) } }, 
	v => v + 2
));
assertType<Array<string>>(runMap(myArray, v => v));
assertType<4>(runMap(
	{ map(cb) { return cb(2) } }, 
	v => v + 2
));

It is important to note that Ezno isn't the first JavaScript type checker that has inferred generics. Hegel infers generics for functions. However I when tried on the above example with its more complicated two levels of inference, Hegel could not figure it out.

The benefit here with generics is that the function can be very expressive and dynamic, but the function still passes off maximum information to the scope where the call occurred.

Free variables (hidden parameters) of functions

The usage of this is a hidden parameter to functions, it is specified by the bounded structure rather than arguments at the call site. Here Ezno treats it as generic but separate from actual parameters:

function getThis() {
	return this
}

assertType<Window>(getThis());
assertType<2>(getThis.call(2));
function getThis() {
	return this
}

assertType<Window>(getThis());
assertType<2>(getThis.call(2));

Variables in parent scopes work similarly to this. Rather than being passed through the call site it instead exists in a parent of the current environment as:

let a = {};

function x() {
	a.doThing();
}

x();
a.doThing = () => console.log("Hello world");
x();
let a = {};

function x() {
	a.doThing();
}

x();
a.doThing = () => console.log("Hello world");
x();

This works using the same generic system that function parameters use. For variables without a type annotation on the variable, it reuses the inferred generic system.

The example above also shows a sneaky feature of Ezno...

Effects/events

One of the problems of JS is that functions can be impure. Impure means it can apply side effects that are not tracked through the returned type.

Ezno tracks side effects that a function may perform:

const data = { x: 0 };

function getFive(obj) {
    obj.x += 1;
    return 5;
}

assertType<0>(data.x);
assertType<5>(getFive(data));
assertType<1>(data.x);
const data = { x: 0 };

function getFive(obj) {
    obj.x += 1;
    return 5;
}

assertType<0>(data.x);
assertType<5>(getFive(data));
assertType<1>(data.x);

Here this function data-wise returns a number with proof of it being equal to 5. But there are additional "effects" that aren't encoded into that return type. When synthesizing functions Ezno tracks mutations through a system in the context/environment called events. Events in a function then get associated with the function referring to the "effects of the function".

The events sequence is sort of a typed intermediate representation and is additionally used for optimisations involving used assignments and such. This has some similarities to SSA. But it is integrated into the checker and is based on types.

Constant functions

Ezno treats function uniquely with a pointer to a function instead of just a "shape" (the same way constant terms and object references work).

We have seen some calculations on operators being calculated at compile time, but the idea carries over to many internal functions. This means it has a direct binding to the function. From this, it can be definitively known that this function is Math.sqrt allowing the following to work:

let x: 2 = Math.floor(Math.sqrt(5));
let x: 2 = Math.floor(Math.sqrt(5));

This applies to more functions than just that in Math. The following also applies to many other functions in the standard library. Here we see effects, this treated as generic and constant functions at play:

const myObj = {
	name: "hello world",
	uppercase() {
		this.name = this.name.toUppercase()
	}
};

assertType<"hello world">(myObj.name);
myObj.uppercase();
assertType<"HELLO WORLD">(x.name);

// Also explicit this calling:
const otherObj = { name: "make me upper" };
myObj.uppercase.call(otherObj);
assertType<"MAKE ME UPPER">(otherObj.name);
const myObj = {
	name: "hello world",
	uppercase() {
		this.name = this.name.toUppercase()
	}
};

assertType<"hello world">(myObj.name);
myObj.uppercase();
assertType<"HELLO WORLD">(x.name);

// Also explicit this calling:
const otherObj = { name: "make me upper" };
myObj.uppercase.call(otherObj);
assertType<"MAKE ME UPPER">(otherObj.name);

JSX

Ezno has support for JSX syntax. A key part of this implementation is that it treats different tags as different tags:

assertType<HTMLParagraphElement>(<p>Paragraph tag</p>);
assertType<HTMLParagraphElement>(<p>Paragraph tag</p>);

It also records more information about the relation of elements to interpolated data and event listeners. (Using the same system that Ezno uses to identify unique objects).

const a = <p onClick={(ev) => {
	assertType<HTMLParagraphElement>(ev.currentTarget);
}>Paragraph tag</p>;
const a = <p onClick={(ev) => {
	assertType<HTMLParagraphElement>(ev.currentTarget);
}>Paragraph tag</p>;

You can think of these as being similar to objects with properties. But in this case, the children binding follow through:

function LazyImage(href: string) {
	return <img href={href} loading="lazy">;
}

const myImage = LazyImage(myInput);
assertType<true>(myImage.getAttribute("href") === myInput);
function LazyImage(href: string) {
	return <img href={href} loading="lazy">;
}

const myImage = LazyImage(myInput);
assertType<true>(myImage.getAttribute("href") === myInput);

Which leads on to the reason for adding these bindings:

The "framework"

One of the biggest uses of JS is with declarative user interface programming.

React is one of the most popular libraries for declarative interfaces. However, its internals are computationally expensive. With Ezno one of the goals was to provide enough static analysis to work with the expressiveness of React but use the information to make it do a lot less work.

"framework" is a temporary name for the plugin written on-top of Ezno, Ezno is not the framework. The "framework" is a plugin using logic from the parser and checker and is entirely detachable.

Eliminating the need for a Virtual Document Object (VDOM)

Firstly, a definition of the VDOM (from my web definitions post):

The virtual DOM is a structure akin to the DOM. It is slimmer and has a subset of the API of the structures defined in the DOM JS spec e.g. HTMLElement. VDOM is a virtual representation of the document, actual DOM references the document (e.g. .click() isn't on VDOM structures).

It is used to add to or update the existing actual DOM/UI.

When the application state changes it recreates the UI by rerunning functions that generate VDOM and use a the difference between the new result with a previous result to make changes/reconcile to the actual DOM/UI. This is a declarative style of programming, where the UI is a map from the state to UI. It is often simpler than manually writing the imperative updates to the document at every state change.

The VDOM isn't free

Before we go into eliminating it we have to deal with the why? As well as the downsides that should occur in the alternative.

Every time an update to the state happens, the runtime needs to rebuild the UI whole tree. This is necessary to find that any of the nodes have changed, but the majority of the new UI hasn't changed and you have to create and store duplicate nodes. The UI can require evaluating expensive calculations that often return the existing value.

Memoization partially solves this, parts of the tree can be wrapped and when compute them it looks for cached versions of the computation. While this solves the recreation time there is additional memory overhead to hold the cached results and hashing and lookup is not free. It is also developer overhead to realise what is static and independent. Extra explicit-ness in the syntax (unless a auto memoising compiler was added), just overall more complicated.

Additionally, to do this handling with the VDOM the library has to be shipped with a reconciler on top of the state handler and the diffing algorithm, all leading to larger JS payloads. There is also the process of starting up the client after SSR. Unless event listener delegation is used ahead of time, when adding event listeners to existing markup VDOM it requires a representation of the whole tree to find event listeners. And even if the event listeners are delegated, a VDOM framework requires creating the existing UI tree to perform updates on an existing server-rendered tree, which requires loading the whole state.

VDOM without the VDOM

One of the questions for this project was 'What analysis do you need to do to use JSX and the full range of JS expressions that React works with, without running into the complexities and execution expensiveness of a VDOM?'.

Because of the tracking of poly-types in Ezno's checker, the values passed around useState can be reasoned with. For example this is what is tracked from the left and right values from a useState call.

const [value, setValue] = useState("");

//                value : any (via dynamic constraint)
//              / [0]
// useState("") 
//              \ [1]
//                setValue : (any) => any
const [value, setValue] = useState("");

//                value : any (via dynamic constraint)
//              / [0]
// useState("") 
//              \ [1]
//                setValue : (any) => any

Because of this, we have three unique types created from this statement that can be tracked.

return <>
	<input onChange={ev => setState(ev.target.value)}>
	<p>{value}</p>
</>
return <>
	<input onChange={ev => setState(ev.target.value)}>
	<p>{value}</p>
</>

Building a compiler to generate imperative calls is then very simple:

Find calls where a function calls the state a function that modifies state (for example useState()[1])
Lookup element points (attributes, children, etc) that are of a type where the base useState is the same useState()[0] (type synthesis stores reverse references on objects)
Replace setState call with direct DOM updates that affect the element points

These updates are relative to the elements in the DOM as these relationships are found during the type synthesis. Therefore the above would generate something similar to:

input.addEventListener("change", ({ target }) => {
	target.nextElementSibling().innerText = target.value;
});
input.addEventListener("change", ({ target }) => {
	target.nextElementSibling().innerText = target.value;
});

Here the type information enables generation of this direct update instead of relying on a VDOM at runtime to find this change.

State objects

const value = useProxy({ count: 2 });

function updateCount(value) {
	value.count++;
}

return <span>{count}</span>
const value = useProxy({ count: 2 });

function updateCount(value) {
	value.count++;
}

return <span>{count}</span>

This sort of thing works in valtio and is the previous implementation in my old framework Prism. This is possible using Proxies at runtime, but suffer from many complications at runtime.

Because of effect tracking in Ezno, it becomes fairly straightforward to find state mutations across functions so these updates can be found and inlined similar to the above. The uniqueness of functions here enables special handling of .map and .push as they are known to be array methods after the type synthesis.

const value = useProxy({ items: [] });

return <ul onClick={() => value.items.push(new Date())}>
	{value.items.map(item => <li>{item}</li>)}
</ul>
const value = useProxy({ items: [] });

return <ul onClick={() => value.items.push(new Date())}>
	{value.items.map(item => <li>{item}</li>)}
</ul>

From this, the logic can work out that calling .push appends a new element to the ul element. Therefore it can skip over needing keys and finding list differences in this scenario and generate a simple .appendChild call.

Universality, adding event listeners and hydration

Most frameworks enable some sort of server-side rendering (often abbreviated to SSR). SSR is the functionality in the framework to generate a HTML string representation of the same UI under a state identically to the result on the browser. It is also used in static site generation where the build tool runs all the requests at build time and saves the results to some sort of static output.

As Ezno knows about bindings and the type interpolated, that information can be used inside the compiler to hydrate the state from the rendered HTML.

const [a, setA] = useState(0);
<button onClick={() => setA(a => a + 1)}>{a}</button>
const [a, setA] = useState(0);
<button onClick={() => setA(a => a + 1)}>{a}</button>

button.addEventListener("click", ({ target }) => {
	target.innerText = parseInt(target.innerHTMl) + 1;
	//                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
	// Here is the reverse expression. Again it knows about types so can invert the implicit cast of a integer to a string
});
button.addEventListener("click", ({ target }) => {
	target.innerText = parseInt(target.innerHTMl) + 1;
	//                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
	// Here is the reverse expression. Again it knows about types so can invert the implicit cast of a integer to a string
});

For upgrading existing HTML, rather than each element registering an event listener (which can result in hundreds of addEventListener calls) it instead can a single one at the top level. The output of the framework is still a work in progress but with the bijective property map, the position of elements works without the notion of components.

createRoot(document.body).render(() => <>
	<button onClick={() => console.log("clicked")}>Log click</button>
	<button onClick={async () => { await fetch("/do-thing") }}>Make request</button>
</>);
createRoot(document.body).render(() => <>
	<button onClick={() => console.log("clicked")}>Log click</button>
	<button onClick={async () => { await fetch("/do-thing") }}>Make request</button>
</>);

possible compiled output:

document.body.addEventListener("click", async ({ target }) => {
	const id = target.getAttribute("data-click");
	if (id === "0") {
		console.log("clicked")
	} else if (id === "1") {
		await fetch("/do-thing")
	}
});
document.body.addEventListener("click", async ({ target }) => {
	const id = target.getAttribute("data-click");
	if (id === "0") {
		console.log("clicked")
	} else if (id === "1") {
		await fetch("/do-thing")
	}
});

Other ideas

Some other ideas that type information can enable:

Server-side rendering out of the JavaScript runtime

One of the problems with JavaScript frontend server rendering implementations is that SSR/"string builders" run on JavaScript which means they are locked into a server runtime like Node or Deno. This is perfectly fine for most things but I think this is generally quite a heavy restriction, which prevents using a lot of cool backend technologies written in other languages like Rust, Python etc. Some tools have embedded a JavaScript runtime into the application (such as rusty_v8). However, joining these two systems together can lose type safety bridge and data often has to be copied into the runtime which seems to void the performance improvements.

With type information, it could allow Ezno to generate some format that is tightly integrated with the server language. This is quite simple to do for string elements and Ezno knows the shape of data and the pointer to every function it could easily transpile some of this stuff. However, it is still up in the air how this would work if the server has to do something more complicated mutating data and I don't have a clear idea of what the format would be that could be used across languages would look like.

Auto progressive enhancement

As Ezno knows at compile time what functions have been bound to certain event listeners plus the internal effects it knows to extract runtime state mutations to the server to add a backup to client-side interactions if JavaScript fails to load.

Plugins and extensibility

Ezno is written in Rust and has several places/hooks for adding additional functionality:

Exterior type safety

Because Ezno treats functions as unique it allows for special handling of functions via Ezno's plugin system.

For example, the fetch function could be overridden and known strings could return a more precise type based on knowledge of what an endpoint returns.

Build tool front-ends

Aside from the CLI, there is a language server plugin (LSP) so you can use it in an editor. For those who don't like the command line build step, there is a half-working Nodejs runner, this means you can integrate the build step into the runner compressing running the source into a single command. For the web, there is a WASM service worker runner that shouldn't need to touch the filesystem during development.

Complexity & wrapping up the features

Enough with the features here is a roundup and some philosophy of the project

Types

Types are an integral part of JavaScript with features like instanceof and typeof. TSC, Flow, Hegel, Ezno and other checkers reason with this runtime behavior ahead of time, which can help find errors before they find themselves. However, unlike JavaScript this ahead of time behavior isn't specified and so each checker has different approaches to what and how it should emit type errors.

Places where Ezno diverges from TypeScript

Given TSC has the largest share in the JavaScript type checking ecosystem, I think it is really beneficial for Ezno to be usable in projects currently using TSC. Many of the ideas from TSC have been included in Ezno, with Ezno going the extra mile in many cases to be more sound so that it can use type-information for build time optimisations.

One of the big things is that Type annotations in source code are treated as restrictions under Ezno and so are not strictly necessary. Ezno prefers to use the JavaScript as the source of truth. From the previous section on events and generic calls, the information needed to generate more accurate results is more complex (and verbose) than simple annotations allow, thus why it is prefered to generate this from the JS code rather than require nnoations.

Also, TypeScript has allows breaking out the type system by the any type to be cast as a specific type (such as string in the following example).

// Passes under TSC
function y(a: any): string {
	return a
}

y(2).slice(2)
// Passes under TSC
function y(a: any): string {
	return a
}

y(2).slice(2)

Implementing any this way makes TypeScript easier to adopt and allows things to compile in weird environments. However for Ezno to do its optimisations this magic any type that has the property of all types without narrowing doesn't quite work.

Don't take Ezno's features as a knock on TypeScript, TypeScript is great. Ezno started as a pet project to reimplement the features of TSC in Rust. But as things started to go well after adding more and more TSC features, it was apparent that optimisations such as the hydration and the reactive systems weren't going to be possible directly following its path. It isn't possible to detect what can happen to the state when called through a term that represents any. TypeScript holes aren't particularly critical if you just want code completions and some basic errors to be caught. They allow the more complex parts to not be blocked by compiler errors. But for doing optimizations a single unknown result can make it impossible.

The hope is with diverging and trying to extend TSC will help in staying up to date with newer TypeScript features (as it will be ahead, rather than behind).

Currently Ezno isn't a feature-complete type checker. There are still a lot of things to still work on. So far the first section goes over the analysis of completely static code. The next step is to apply these ideas to more dynamic structures. There are also some ideas not mentioned here to prevent this post from being too long 👀.

Frameworks

The output of the framework is still a work in progress. There are still a few things to add to the type checker that are necessary to get "the framework" plugin to work in edge cases. Thus no benchmarks or definite output in this post. Most benchmarks show hand-written JavaScript code to be the most efficient so the idea is to take the syntax and information about it to squeeze it into the closest of the hand-written forms. The performance gains from all these optimisations are probably unnoticeable. This is an exercise in attempting to get to the least amount of code to run to make a page interactive.

Some frameworks abstract through libraries and the structure of functions. Some compiler-based frameworks abstract through syntax. Ezno's framework is unique it is based on data and the semantics of the program above what is visible from the syntax.

Ezno today

I think it is fun short and quirky and most importantly, not taken on package registries.

I want to keep things moving but slowly. I think incomplete projects are good as they have space to add additional features and improve. Some projects I see which seem closer to their goal have less space to improve. I think some tools are built too quickly and I don't want Ezno to fall into that category.

~~No demo binary out yet, need to finish off some things on more advanced events to get some of the cooler demos to work. Hoping for something demonstrable before the end of this year 🤞~~

Ezno and its checker are now public as of 2023

Definitions of web terminology

2022-03-31T00:00:00Z

Rendering

The act of turning data or state into some representation of HTML.

Server-side rendering (SSR)

A process that produces a string/text buffer of the HTML format by a server (a machine that is not the client). Note that this can be stored in buffers on a server as a cache or can be saved to files as static site generation.

Client-side rendering (CSR)

A string/text representation of HTML or HTMLElements that is created locally on the client (the same CPU, this includes web workers and service workers).

Hydration

Hydration is the code that associates the state of an item with an existing object. On the web, this refers to creating state in JS that matches server-side rendered DOM, which then enables client-side interactivity. It has origins in database and originates from the hydrate project from 2006.

Partial hydration/Island architecture

Partial hydration (for a full page) is only hydrating certain dynamic elements (not static trees). The islands refer to the dynamic trees.

Partial hydration is a form of dead code elimination, where the code being removed is anything to do with static UI. This includes static components render methods and any dependencies those render methods pull in. If the state is also serialized as a JSON blob then partial hydration can remove data used by static trees.

Partial hydration is an architecture, not a feature. Partial hydration/Island architecture is unrelated to progressive enhancement.

Progressive hydration/Lazy hydration

Doing the process of hydration on interaction with a component or some other time after the page load.

Progressive enhancement (PE)

The act of server rendering markup that has functionality using HTML features such as:

Forms with endpoints
Anchor tags with href

The above features are available without JS running on the client. A server should be ready to receive the browser's requests and do stuff.

Progressive enhancement can be implemented in any framework that supports server-side rendering. Although some frameworks have helpers for making this easier or the cow path. Additionally, the hydration step can then add event listeners to override the default browser functionality. progressive enhancement is not partial hydration.

Note that not every interaction can be implemented using the HTML features (e.g. mouse drawing on a canvas) so JS is required in many scenarios and often doing it in JS has a better experience. Forms and anchor tags both do full page reloads which can disrupt the state present on the client (e.g. other input contents, background audio).

But doing PE is better than not doing PE. The site should be as functional as it can be because JS (the hydration) may have not run yet, at all or may have failed.

Design system

A collection of colour themes, component designs, layouts, and assets encompassing brand image and packaged up to be easily reusable throughout the site.

Hot module reloading/swapping

When changes happen during development it patches the changed functions rather than reloading the whole content and state.

Dynamic rendering

The server renders pages with data on demand.

HTML Frames

Sections of which content is purely server-side rendered. Note that server components may or may not be HTML frames if they don't serve HTML strings.

Static site generation (SSG)

Generation of static HTML through server-side rendering (or some bad methods that do client-side rendering and then capture the result on a headless browser). Normally done at build time. Note that SSGs can have dynamism through client-side rendering. If a tool has server-side rendering then that mechanism can be used to store the HTML content in a file to implement SSG.

Incremental static generation (ISG)

Similar to a static site generation. Where a static site generates on build/deploy. Incremental static generation is linked to a timed interval or a hook on the change of a data source. The hook fires an event that generates new pages to reflect the new data.

Time to interactive (TTI)

The time to it takes to add event listeners with most of the functionality ready.

Event handlers are registered for the most visible page elements
The page responds to user interactions within 50 milliseconds

First contentful paint (FCP)

The time to display content from when the page starts loading (e.g. server initially responds, so this does not include the time to establish a connection). If SSR this is the time it takes to produce the HTML string and if doing purely client-side rendering then the time it takes for the JS to start running and produce the elements. This also includes initial layout working and image rendering.

Time to first byte (TTFB)

The time it takes for the server to initially respond. If streaming, this is the time for the first chunk. If not (aka buffering) this is the time for the content to be prepared and sent (aka a full SSR). Normally a measure of the hosting server rather.

No-JS/zero JS

Something with no JS running ever. This includes 3rd party scripts. Again most pages require some form of JS, something that runs no JS is not necessarily better.

Sprinkles

Using JS to add interactivity to certain parts of a server-rendered UI in small amounts. (This is not progressive enhancement as it doesn't require implementing server functionality)

3rd party scripts

A script that is written out of house. Examples include Google Analytics, Google Tag Manager, etc

Static trees

A tree that does not change/depend on variable data

const static_tree = <h1>Hello</h1>;
const still_static = <div>{*constant variable*}</div>;
const not_static = <h2>{new Date()}</h2>
const static_tree = <h1>Hello</h1>;
const still_static = <div>{*constant variable*}</div>;
const not_static = <h2>{new Date()}</h2>

not_static is dependent on a variable result and thus is a dynamic tree.

Re-render

For VDOM or systems that need to recalculate trees after the result of an action. This term is given to the calculations for recomputing UI. Later the result of the re-render requires diffing to efficiently update the new UI.

Fine-grained reactivity

Reactivity in which knows about parts of the states and only does work in those areas.

<h1>{title.toUppercase()}</h1>
<p>{content}</p>
<h1>{title.toUppercase()}</h1>
<p>{content}</p>

E.g. changing content should not result in calculating title.toUppercase()

This is to do with partial updates to the state. Something that skips static trees is not really fine-grained reactivity.

Note that fine-grained reactivity may include re-rendering sections that are dynamic under the state.

Server component

Any component where its content is produced on the server either in HTML or an intermediate format.

Memoization

The process of caching function return values against the inputs. If a function takes a long to compute the result and is rerun a lot then this can speed up getting the result as it takes a map lookup rather than a re-computation.

An auto-memoization compiler can wrap function calls with a cache lookup and storage.

Note that this is an optimization at call sites, this can be avoided via rearranging when data is calculated and passed through.

This technique incurs memory overhead due to the need to store all results of the function. And for many cases a map lookup can be slower than just doing the operation.

Actions

Something that mutates a specific part of the state.

Effects

Results of those actions. e.g. changing a value may require updating the content element in the templating interpolation.

Diffing

Finding differences between an existing representation and a new representation.

Diffing techniques do not always apply to VDOM. Diffing can be done on structures that do not look like DOM. Such as a flat list.

Produces a diff/difference that can be used for reconciliation.

Conciliation/reconciliation

This applies to virtual DOM and other representations e.g. lists.

It is the act of taking the results of the diff and updating the UI. The diff should describe the minimum amount of work to update the UI).

CSS in JS

Some notion of writing styles in JS. Normally via object literals that look like CSS syntax. Unsure whether this covers <style> JSX tags and template literals...

Frontend

Frontend is what is interpreted by the client. Public and visible to all. Includes communication with backend (but not implementation).

Backend

Something that does not run on the client. Owned by an operator, distributes data and effects across clients. The backend includes the serving of content and HTTP responses etc

Full stack

The combination of frontend and backend. Full stack knowledge is knowing both sides of the network. A full-stack framework has features spread across frontend and backend

Single page application (SPA)

A page that does not use the browser's built-in navigation to do page transitions. It does not mean that there is only one page, only that the browser internally thinks it is on the same page.

Can be faster as only have to update regions between pages, and can retain state between navigations. Implementations should be using the history API so that the browser's back buttons still function. New page contents can be generated using client-side rendering or by retrieving and injecting server-rendered content.

A SPA can be server rendered initially and this architecture makes it simple to build a PWA.

Multi page application (MPA)

A page of which links cause inbuilt browser navigation. Pages are exclusively server-rendered (but parts of them can be changed via the client).

Progressive web application (PWA)

This encompasses a lot, but the main points are that it is built using web technologies but can do the following:

Installable (act like a native app)
Work offline (functionality does not require talking to a server all the time, some content is stored on the device)
Doesn't have to but uses several native APIs: camera, clipboard, background fetch, push notifications

Static

Cannot change.

Dynamic

Can change.

DOM (Document object model)

The API for HTML elements. Every HTML element has some attributes and some children either being more elements, text, or comments. DOM elements can also interact with methods like .click().

Shadow DOM

A special form of DOM that is encapsulated inside the element. The internals of shadow dom are isolated from the whole DOM so that outside JS cannot reference and CSS cannot affect. CSS defined internally is scoped to the internal tree.

Virtual DOM/VDOM

The virtual DOM is a structure akin to the DOM. It is slimmer and has a subset of the API of the structures defined in the DOM JS spec e.g. HTMLElement. VDOM is a virtual representation of the document, actual DOM references the document (e.g. .click() isn't on VDOM structures).

It is used to add to or update the existing actual DOM/UI.

Universal JavaScript/Universal rendering

Running JavaScript produced that is derived or is the same source on both the client AND the server.

Isomorphic JavaScript

Same meaning as universal JavaScript.

Use of this should be discouraged as the (proper) definition of isomorphic in category theory doesn't make sense here.

Meta framework

A framework that is built upon one or more existing frameworks and wraps functionality. For example, nextjs that extends React.

Templating language

A language that can describe how to build some form of markup.

Imperative templating

A template language that has imperative notions of a declarative source.

Streaming

Streaming is incremental sending parts enabling work to start happening without the whole of the resource being present. e.g a streaming renderer can start returning results before the whole thing has been rendered. Streaming hydration can start hydrating nodes before all the nodes are on the client.

Static analysis

Something that is statically analyzable is something of which behaviour can be worked out ahead of time. It should be noted that some things that are deemed not to be statically analyzable can be made statically analyzable by introducing constraints on what can be written. It should also be used with caution as some things named under statically analyzable are but whose implementation is incredibly complex to build.

Markup

Some kind of language that is centered on content first. Markdown, HTML, and YML are based on content.

Compiler

A program that:

Parses input into abstract syntax trees, concrete syntax trees, or some other source-based IR
Transforms IR
Returns some evaluatable result, or a collection of errors found

"compiling" is a compiler at work.

Transpiler

Similar to a compiler it is a source-to-source compiler. Source to source meaning the output is in the same format as the input. A transpiler is a subset of compilers.

"transpiling" is a transpiler at work.

Intermediate representation (IR)

A more abstract representation of the source, may not be reversible to the original source. e.g. operation canonicalization.

Serverless

Non-centralized computation. Similar to a pure function (without side effects) these should be small map-like functions.

Note this is still run on serverless just that it abstracts away a lot of the behaviours of centralized server computations.

API

The definition/interface for interaction with something.

Headless browser

A browser that is controlled by a server rather than a user. Examples of headless browser tools include puppeteer, selinum and playright.

Tooling

A single or a collection of programs that are used to build a program.

Hoisting

The use of functions before their definition in the source program.

Framework

Something that acts as the entry point to a program. The framework interprets and operates over user code.

There are several kinds:

Backend/HTTP frameworks
Frontend frameworks
Parser frameworks
Test frameworks
Benchmark frameworks

HTTP framework

An abstraction for a server in receiving HTTP requests and returning HTTP responses. Examples: express, oak and axum.

Frontend framework

Code that only executes on the frontend

Fullstack framework

Code that only executes on the frontend and backend

Library

Something that exposes functions that can be called and returns results.

Primitive

Something of which its internals cannot be read.

Bundling

Concatenation of source code from multiple sources and files into one or more files.

Dead code elimination (DCE)

Finding code that is never run or has no effect and making sure its representation doesn't end up in the final output.

Tree shaking

Tree shaking is a subset of DCE that mostly refers to removing top-level function declarations (from the abstract syntax tree, which is what the tree part refers to).

(whitespace) Minification

Remove unnecessary whitespace (new lines, tabs, spaces) from the source.

Infrastructure

The whole operation or managing and running the program.

Standard

An API that is formalized in a specification and implemented by other parties.

Type checking

Validating that source code lines up with type definitions.

Type annotation

A piece of syntax that associates a type with a term.

Type inference

Identifying a type without using information from a type annotation.

Immutable

Cannot change

Mutable

Can change

Cache

A map like data structure that may or may not contain an already processed or downloaded asset

Prism Compiler 1.5

2021-03-22T00:00:00Z

Prism 1.5 is out now. This post goes over the server component craze, frontend frameworks on the backend, why frameworks and the future of Prism.

Prism is a experimental framework for building web apps which takes declarative templates written in HTML, CSS and JS (similar to Svelte and Vue) and compiles them into small & efficient client and server bundles. It is a research project into some ideas of what I thought could be more efficient isomorphic frameworks.

"Server components"

Last December there were two big announcements in the world of server rendering. The first was React server components which enables components VDOM content to be rendered on the server. The other was hotwire, which is a set of libraries for adding more interactivity on the page through communicating with the server.

These two announcements sparked ideas around "server components" which is a different to the status quo in standard data loading & rendering flow:

Standard AJAX flow:

Read request → JSON → parse → build DOM → append to node

HTML (or VDOM) over the wire:

Read request → append request body as content to a node

Both are interesting ideas, but looking at both notice that they have constraints and still areas to improve performance. For Prism, I wanted to capitalize on some of the benefits of rendering on the server but in a more flexible and low runtime way.

Prism server components

All frameworks are based on components so to start we can send the following HTML down the wire:

<story-preview>
    <div class="buttons">
        <button disabled>&#9650;</button>
        <button disabled>&#9660;</button>
    </div>
    <div>
        <h2>
            <a href="https://chartscss.org">Charts.css</a>    
        </h2>
        <span>
            882 points | by
            <a href="/u/pspeter3">pspeter3</a>    
            <span title="Sun Mar 21 2021 10:37:57 GMT+0000">| 1 days ago |
            <a href="/i/26494819">135 comments</a>
        </span>
    </div>    
</story-preview>
<story-preview>
    <div class="buttons">
        <button disabled>&#9650;</button>
        <button disabled>&#9660;</button>
    </div>
    <div>
        <h2>
            <a href="https://chartscss.org">Charts.css</a>    
        </h2>
        <span>
            882 points | by
            <a href="/u/pspeter3">pspeter3</a>    
            <span title="Sun Mar 21 2021 10:37:57 GMT+0000">| 1 days ago |
            <a href="/i/26494819">135 comments</a>
        </span>
    </div>    
</story-preview>

(This is an simplified) example from HN prism

This example would have been rendered on the server, interpolating data from some source, database etc. The browser then parses the response body and renders the tree in an optimised procedure without running JavaScript. The links are readily clickable and the buttons have a disabled attribute considering they don't have any functionality until a JS event listener is attached.

While this is great there are several scenarios where you want to do some work on the client with the components the server responded with. For example:

// Upvote post (needs previous value)
this.data.score += 1; 

// Saving title in indexedDB
objectStore.add(this.data.title); 

// Making request with data
fetch(`/upvote-post`, { 
    method: "POST", 
    body: new URLSearchParams({"postID": this.data.postID})
})
// Upvote post (needs previous value)
this.data.score += 1; 

// Saving title in indexedDB
objectStore.add(this.data.title); 

// Making request with data
fetch(`/upvote-post`, { 
    method: "POST", 
    body: new URLSearchParams({"postID": this.data.postID})
})

To run the above the client needs to know the value of the data used to render the component's markup. However, given the server sent markup it's not known what the data used to render that page on the server looked like. Neither is there any way of guessing, there are eight numbers in the response any of those could be the value for score.

So the answer by many frameworks is to serialize the data used to render that component on the server and include it in the response. The component's associated data is then deserialized through JSON.parse. Here is an example of a response with the serialized data:

<story-preview>
    <div class="buttons">
        <button disabled>&#9650;</button>
        <button disabled>&#9660;</button>
    </div>
    <div>
        <h2>
            <a href="https://chartscss.org">Charts.css</a>    
        </h2>
        <span>
            882 points | by
            <a href="/u/pspeter3">pspeter3</a>    
            <span title="Sun Mar 21 2021 10:37:57 GMT+0000">| 1 days ago |
            <a href="/i/26494819">135 comments</a>
        </span>
    </div>
    <script type="data">
        {
            "url": "https://chartscss.org/",
            "title": "Charts.css",
            "score": 882,
            "by": "pspeter3",
            "time": "Sun Mar 21 2021 10:37:57 GMT+0000",
            "id": 26494819,
            "descendants": 135
        } 
    </script>
</story-preview>

<story-preview>
    <div class="buttons">
        <button disabled>&#9650;</button>
        <button disabled>&#9660;</button>
    </div>
    <div>
        <h2>
            <a href="https://chartscss.org">Charts.css</a>    
        </h2>
        <span>
            882 points | by
            <a href="/u/pspeter3">pspeter3</a>    
            <span title="Sun Mar 21 2021 10:37:57 GMT+0000">| 1 days ago |
            <a href="/i/26494819">135 comments</a>
        </span>
    </div>
    <script type="data">
        {
            "url": "https://chartscss.org/",
            "title": "Charts.css",
            "score": 882,
            "by": "pspeter3",
            "time": "Sun Mar 21 2021 10:37:57 GMT+0000",
            "id": 26494819,
            "descendants": 135
        } 
    </script>
</story-preview>

The problem is now each response is now larger due to the same data now existing both in the markup and in the JSON blob (as shown with the highlighting). This problem is often referred to as the double data problem. This is especially a problem in VDOM and render to hydrate based frameworks which require all the data to start 1.

A different approach to getting state

As we have seen just looking at the HTML response it is impossible to pull data from it. But now if I show you the template that was behind what was used to render story-preview's HTML then it becomes understandable where those values were interpolated:

<template>
    <div class="buttons">
        <button @click="upvote">&#9650;</button>
        <button @click="downvote">&#9660;</button>
    </div>
    <div>
        <h2 #if="url">
            <a $href="url" target="_blank" rel="noopener noreferrer">{title}</a>
        </h2>
        <h2 #else>{title}</h2>
        <span class="details">
            {score} points | 
            <a relative $href="`/u/${by}`">by {by}</a>
            <span $title="time"> | {timeFromNowToString(time)} ago | </span>
            <a relative $href="`/i/${id}`">{descendants} comments</a>
        </span>
    </div>
</template>
<template>
    <div class="buttons">
        <button @click="upvote">&#9650;</button>
        <button @click="downvote">&#9660;</button>
    </div>
    <div>
        <h2 #if="url">
            <a $href="url" target="_blank" rel="noopener noreferrer">{title}</a>
        </h2>
        <h2 #else>{title}</h2>
        <span class="details">
            {score} points | 
            <a relative $href="`/u/${by}`">by {by}</a>
            <span $title="time"> | {timeFromNowToString(time)} ago | </span>
            <a relative $href="`/i/${id}`">{descendants} comments</a>
        </span>
    </div>
</template>

From story-preview on hackernews-prism

The template reveals what is interpolated where. Comparing it to the previously shown response you can begin to see where values match up to the server response. For example, there is a variable/property named score which comes before "points".

Knowing how the data maps in the template, a compiler could generate JS getters for retrieving data from the server-rendered DOM.

In the above example, the title could be retrieved at runtime with the following getter:

get title() {
    return component.querySelector("h2").innerText 
}
get title() {
    return component.querySelector("h2").innerText 
}

Using this method, state could be brought into JS from the server-rendered content making a JSON blob redundant. No more JSON state leads to a reduction in the bytes sent over the wire.

And this is how the hydration system in Prism works. The compiler builds a table of bindings in the template. From that, it can generate code similar to the above statement. This same binding table is also used to generate set bindings for reactivity.

In the following HN clone you can see HTML come down the wire for rendering story-preview components:

From this, you can see the story-preview components content is coming from the server as HTML. The rendered markup only contains one instance of the component's title, score and url properties thus no double data. temp1._d shows what data the component was given during instantiation but when JSON.stringify(temp1.data) it evaluates every property by looking it up in the components server rendered DOM. The comments () in the response are to break up text nodes.

Not only is the state available to the component but is also public to other components. Running JSON.stringify(document.querySelector("index-page").data, 0, 4) should see an object with an array of stories. Those stories exist on the individual components but modifying externally is permitted: document.querySelector("index-page").data.stories[2].title = "Hello World"

The resolved data is being pulled from the HTML content of the component when evaluated. I called this implementation JIT hydration.

Using this, Prism server components can send standard HTML down and become interactive on the client. A component can mark that its content should be from the server via the @RenderFromEndpoint decorator which takes a parameterised URL that points to an endpoint that returns the content of the component.

@RenderFromEndpoint("/story-preview/:id")
export class StoryPreview extends Component<IStoryItem> {
    ..
@RenderFromEndpoint("/story-preview/:id")
export class StoryPreview extends Component<IStoryItem> {
    ..

Which is compiled down to:

render() {
    fetch(`/story-preview/${this.data.id}`).then( async (resp) => {
        this.innerHTML = await resp.text(); this.handleEvents?.(true)
    })
}
render() {
    fetch(`/story-preview/${this.data.id}`).then( async (resp) => {
        this.innerHTML = await resp.text(); this.handleEvents?.(true)
    })
}

No streaming yet 4

One nicety of this is that it associates data under an identifier. Prism (and most other frameworks) work by a parent component resolving data and then rendering a child component with that data. With this system, components are self-aware and rerender themselves on updating id:

The HTML is coming from a Prism compiled toString function which given data returns a string of the concatenated component markup. All that is required is hosting an endpoint that calls the generated function and returns the response:

// Import prism compiled function
import { renderStoryPreviewContent } from "./out/story-preview.prism";

app.get("/p/:id", (req, res) => {
    res.send(renderStoryPreviewContent(getStoryPreview(req.params.id)));
});
// Import prism compiled function
import { renderStoryPreviewContent } from "./out/story-preview.prism";

app.get("/p/:id", (req, res) => {
    res.send(renderStoryPreviewContent(getStoryPreview(req.params.id)));
});

As Prism only builds the functions you can build the backend using any node framework or alternatively use them on a serverless function platform

The server-generated function and hydration system existed for the initial SSR. With the addition of server components, the efficiencies in these methods now extend to the lifecycle of the page. This system is not a two-way binding system as the retrieval is only done once. Two-way bindings often refer to inputs whose value fires updaters when the value of an input changes.

Events are attached when the custom elements connected callback fires. The compiled implementation finds elements and calls addEventListener. Unlike others, Prism does not do any sort of rerendering to add event listeners. This results in super quick TTI.

Prism already had small bundle sizes. Without server components, the total uncompressed bundle is 17kb and after converting story-preview and story-page for their content to be rendered on the server it comes to 16.08kb which is 4.76kb after GZIP. The saving of around ~1kb is around removing server loading logic and (some) of the render methods for the components. The bundle could be ~25% less if Prism could tree-shake the reactivity logic and minify identifiers.

Some other features Prism with its hydration implementation (called JIT hydration)

Getting values from the client is lazy. The get logic is only called when the value is evaluated.
Getting values on the client is done on a per-property basis. title can be in the JS runtime but not time.
Caches the returned value to not be a call to the DOM every time.
Even though the DOM is made up of strings, Prism can convert various types. This is why type declarations are required 2.

Using `@RenderFromEndpoint` has the following benefits

Skipped the parse cycle on the JSON returned from the HN REST API and the runtime creating the nodes on the client which should be a little bit faster.
Reduced the client logic for getting the data and rendering nodes.

Comparisons

Compared to React server components

Rendering stateful components (RSC does not allow for stateful components)
Significantly smaller JS size. React starts at 133kb uncompressed, Prism starts at 2kb of JS
Does not require JS on the backend (see next section)

One thing I will give React server components is the ability to write backend logic in line with the server components which is kinda neat.

Compared to Hotwire

Prism defaults to client rendering. Hotwire has no client rendering implementation,
Significantly smaller JS size. Stimulus is 77.4kb and Turbo is 80.4kb.
Smaller response payloads. Stimulus attribute-based logic is sent down on every response. On the other hand, Prism compiles logic into the JS bundle so the logic size is constant and can cached between requests.
Stimulus seems to suffer from the double data problem with its data-*-value attributes.

On JSON vs HTML

So the good thing about sending HTML down is that it can be readily placed into the tree without a transformation step. Comparing the sizes: JSON includes the keys whereas Prism's hydration system encodes the key mapping into the hydration code generation, which is constant for any incoming data. HTML is a data language although data is nested within a bunch of UI markup. For example, on every story-preview it has to send two buttons for each component although with JSON only the raw data is sent and the buttons are added via constant bundle code.

Looking at HN front page the average JSON size for story-previews was around 220bytes whereas the average inner HTML size was around 600bytes. Thus making HTML around 2.5x larger than its equivalent JSON representation.

The figures are a little skewed against HTML as Prism includes identifier classes, which could be reduced if Prism moved to an index-based element lookup system. The size factor varies between components depending on how much of the template is made of data compared to static markup. There is also the fact some of the literal expressions cannot be reversed so a little bit of extra data is added 3. These figures are ignoring compression which may have a disproportionate effect between the formats and may close the size gap. But both formats aren't great data formats for small efficient data flow. Inspired by serde, I have some opinions on how compilers and strong types could be used for making more efficient serialization and deserialization.

On JIT hydration vs Partial hydration

One improvement to full hydration is techniques partial and progressive hydration. Partial hydration seems to benefit render-to-hydrate frameworks (which Prism isn't) by rerendering only islands (rather than the whole page) to add interaction. Partial hydration is difficult to implement though as it is difficult to know what portions are interactive and stateful. And although static regions are now ignored dynamic regions still suffer from the double data and rerender issue. In this case for HN Prism this means the biggest components story-preview and story-page aren't any more optimized.

Progressive hydration is incrementally making portions interactive rather than waiting for everything to be processed before event listeners are added.

However in Prism whether a component is stateful or not it still doesn't send a JSON blob or rerender. The state is ultra partial and progressive considering properties are only retrieved when they are being evaluated and only the single property of that object is hydrated in. I think its reversible hydration system and the code generation around the data is the only way to solve the double data problem for stateful components.

Frontend frameworks on the backend

One of the arguments behind Hotwire is that its system works for server-rendered sites built in languages other than JS. This is generally a problem with all front-end frameworks. React, Vue, Angular and Svelte all have some API to render their templates to a string but they are all restricted to the JS language. This is a big gap as there are lots of other backend frameworks and tools for languages not in JS.

So for Prism, I added the ability to target Rust for its server-side rendering output. The HN demo is written as Rust + actix-web server. Speed was a focus for this site which is why ActixWeb was chosen as it is one of the fastest backend frameworks. The deserializing from the HN REST API is done with Serde which is renowned for its speed. And of course, the compiler-based GC heavily optimized Rust language base is key to these results. Comparing ActixWeb against Express, Actix excels in the server framework benchmark. The biggest standout of this article though is that Actix is 6x more efficient than nodejs. This is great for lowering server running costs and most importantly a sixth of the electricity 🌲🌳. Additionally, Prism server-side rendering is also available for nodejs and Deno.

For those building a Rust REST backend to a client-side rendered site, it means you can add SSR to it without having to deal with both a node and a Rust server and communication between the two.

Yep, that's right Rust server-side rendered web components, never thought you'd see those words together.

All that is required to render a Prism component/page is to wire up the generated method to an endpoint:

// Import Prism generated method:
use templates::story_page_prism::render_story_page_page;

#[get("/i/{storyID}")]
async fn story_page(web::Path((story_id,)): web::Path<(i32,)>) -> HttpResponse {
    let result = api::items::get_story(story_id).await;
    if let Ok(post) = result {
        HttpResponse::Ok()
            .content_type("text/html")
            .body(render_story_page_page(&post))
    } else {
        HttpResponse::InternalServerError().finish()
    }
}
// Import Prism generated method:
use templates::story_page_prism::render_story_page_page;

#[get("/i/{storyID}")]
async fn story_page(web::Path((story_id,)): web::Path<(i32,)>) -> HttpResponse {
    let result = api::items::get_story(story_id).await;
    if let Ok(post) = result {
        HttpResponse::Ok()
            .content_type("text/html")
            .body(render_story_page_page(&post))
    } else {
        HttpResponse::InternalServerError().finish()
    }
}

The fact that React server components were restricted to a node backend was discussed in the comments of the RFC. Without embedding v8 and making calls it looks pretty distant that React SSR could be fully functioning on non-js backends. The difficulty with embedding v8 is that you lose strong typing from Typescript.

I should also mention WASM and Rust-based "frontend-frameworks" yew, percy & seed here. They look very interesting, make effective use of procedural macros, have Rust SSR support and are the first real way to write DOM-based client-side code in a different language to JavaScript (other than languages that compile to JavaScript). However, WASM has a few disadvantages for the client-side apps. Firstly the size of the WASM bytecode seems to be larger than if the logic was written in JS. I don't quite know the specifics of WASM bytecode but from machine bytecode know that things like generic implementations, bundling standard library and inlining bumps up the size. All of which serving JS doesn't suffer from. Yew states that its example bundle is ~100kb which is similar in size to React which isn't great for slower connections. Also, the component sizes are likely to be larger than if they were written in JS. Yes, their bundle is faster to parse and compile but TTI is hampered from loading over the network. Secondly, they seem to use VDOM and diffing (rather than a compiled reactivity approach) so while being close to the metal they still generally do more computation vs direct compiled setters. That also means they probably suffer from the double data issues that Prism escapes. Also, the fact that WASM can't call arbitrary JS methods and instead has to be passed them. So fetching etc still requires writing JS and passing references to the instance. Cool at the moment for canvas rendering etc but generally not the silver bullet for JS-based frontend frameworks.

For Prism the most promising feature with WASM is the available runners. If Prism could compile SSR functions for WASM then it could be used with the Python runner and wouldn't have separate compiler outputs. Untested but I think Prism's rust output as binary could be called from the Python runtime or any other language that can call C-like code.

Reflection on Prism

With Prism I took a lot of the problems around frontend frameworks today into account with the design. I hope I at least made a dent in some of these issues:

Double data from SSR
SSR in languages other than JavaScript (in this case Rust)
The large size from a frameworks runtime

At this point, it seems necessary to mention why frameworks? and why plain JS or no build step is insufficient in many cases for production sites.

Why frameworks?

Frameworks generally implement a single declarative way to mark that this variable/data/state is interpolated here. Reactivity mechanisms ensure that the view is always up to date with the current value of the variable/data/state. Generally HTML doesn't have a way to express a binding with JS. So updating the view is done imperatively:

customElements.define(
    "counter-component", 
    class extends HTMLElement { 
        counter = 0; 
        
        connectedCallback() {
            const h1 = document.createElement("h1");
            h1.innerHTML = this.counter;
            const button = document.createElement("button");
            button.innerHTML = "+";
            button.addEventListener("click", () => {
                this.incrementCounter();
            });
            this.append(h1, button);
        }
        
        incrementCounter() {
            this.counter += 1;
            this.querySelector("h1").innerHTML = this.counter;
        }
    }
);

const counterComponent = document.createElement("counter-component");
document.body.append(counterComponent);
customElements.define(
    "counter-component", 
    class extends HTMLElement { 
        counter = 0; 
        
        connectedCallback() {
            const h1 = document.createElement("h1");
            h1.innerHTML = this.counter;
            const button = document.createElement("button");
            button.innerHTML = "+";
            button.addEventListener("click", () => {
                this.incrementCounter();
            });
            this.append(h1, button);
        }
        
        incrementCounter() {
            this.counter += 1;
            this.querySelector("h1").innerHTML = this.counter;
        }
    }
);

const counterComponent = document.createElement("counter-component");
document.body.append(counterComponent);

(excludes cleanup)

Here the interpolation of this.counter is done imperatively and due to this, it requires writing the interpolation logic twice. Writing twice is a little time-consuming but the worst effect is that the logic is split up. If the first part is modified for example h1 tag is changed to h5 then the second part is broken and it's not clear for a linter to pick this up. The desync issue here is manageable but when you have lots of components with lots of interpolation spread across a large project with lots of contributors it gets difficult to manage. There is also the fact that the imperative calls are quite distant from the declarative design of the HTML language and reading & processing the above is more difficult. However, the lack of interpolation synchronization only gets worse...

Universality/SSR

The reactivity issue is further amplified when work is shared between the client and server. For this example, a "post" is sent down with some interpolated data.

server.js:

res.send(`
    <h1>${postTitle}</h1>
    <button id="upvote-button">Upvote post</button>
    <span id="upvotes">${upvotes}</span>
`);
res.send(`
    <h1>${postTitle}</h1>
    <button id="upvote-button">Upvote post</button>
    <span id="upvotes">${upvotes}</span>
`);

And after some action on the front-end incrementing this.upvotes. Updating the text of the span is done with:

client.js:

document.querySelector("span#upvotes").innerText = this.upvotes += 1;
document.querySelector("span#upvotes").innerText = this.upvotes += 1;

This is not great because there is a loose reference to span#upvotes. The server response may be changed to use p instead of a span and now the querySelector call returns null. It is not easy to find the issue and often ends up in a "spot the difference" or "where's Wally" scenario across separate files. I have found this on large projects where I go to change or add a button and now have to find out what code was relying on that button and what that affects. This problem is amplified when there are tens of pages and hundreds of places where things are interpolated and events are connected.

Checking

There is also the fact that the above server code is a raw string literal. It does not check if it is valid HTML at compile time (some templating languages may do, not quite sure) so I have often lost time after writing something like <h1 ${someX}</h1>. With Prism it will always concatenate to valid HTML and as a compiler, it also catches syntax errors when parsing templates. The Svelte framework takes this checking a step further linting the template with rules to ensure accessible HTML. This is only really possible with the template DSL of Svelte as a template literal can still be valid without knowing what is interpolated.

The other thing Prism does is add the disabled attribute to buttons with events which it then removes on adding event listeners. Also, the above snippet is susceptible to XSS scripting attacks. Prism (and other template languages) wrap all interpolations in escape-safe calls. The other thing is Prism auto generates non-clashing identifiers. The incrementing example above would break if I added a new element on the page with an identifier #upvotes.

Lists

With lists, you may want to render the first 10 items in the server responses and later add more in an infinite style way (the same way Twitter and Instagram feeds work). So on the server, I may write a function that renders an element of a list:

function renderListItemToString(item: IPost): string { .. }
function renderListItemToString(item: IPost): string { .. }

However now on the client if I wanted to append a new item the renderListItemToString is only available on the backend (not the frontend). With a framework that has or compiles multiple functions depending on runtime the same list item elements can be generated on both the frontend and backend. I guess this is an advantage of server components and Turbo where the frontend function is an alias for calling the same function on the backend under the same source.

Single source

So frameworks implement some sort of single source. For example in Prism:

<template>
    <h1>{postTitle}</h1>
    <button @click="upvotePost">Upvote post</button>
    <span>{upvotes}</span>
</template>

<script>
    public interface PostData {
        postTitle: string, 
        upvotes: number
    }

    @WithCapacity(100)
    class Post extends Component<PostData> {
        upvotePost() {
            this.data.upvotes += 1;
        }
    }
</script>
<template>
    <h1>{postTitle}</h1>
    <button @click="upvotePost">Upvote post</button>
    <span>{upvotes}</span>
</template>

<script>
    public interface PostData {
        postTitle: string, 
        upvotes: number
    }

    @WithCapacity(100)
    class Post extends Component<PostData> {
        upvotePost() {
            this.data.upvotes += 1;
        }
    }
</script>

The template is declarative. It abstracts on the imperative document.createElement and attachEventListener calls. The template is much more akin to HTML and understanding the structure of this component is more accessible. The span <-> upvotes binding is only written once. And so if span was changed to p there are no other handwritten references of this binding and compiling would take care of updating all references to span with references to the p element. For full reactivity and JIT hydration, Prism will take the single source and generate different code to handle for each case. These would be tricky to manage if written manually. For example, the upvotes binding eventually ends up in four places:

client.js:

// Initial render
render() {
    this.append(.., h("span", {class: "p120"}, 0, this.data.upvotes))
}
// Reactive bindings
bindings = {
    ..
    upvotes: {
        get() { return parseInt(this.getElem("p120").innerHTML); },
        set(value) { this.getElem("p120").innerHTML = value; },
    }
}
// Initial render
render() {
    this.append(.., h("span", {class: "p120"}, 0, this.data.upvotes))
}
// Reactive bindings
bindings = {
    ..
    upvotes: {
        get() { return parseInt(this.getElem("p120").innerHTML); },
        set(value) { this.getElem("p120").innerHTML = value; },
    }
}

server.rs:

pub struct PostData {
    postTitle: String, 
    upvotes: f64
}

fn render_post(post: IPost) -> String {
    let mut acc = String::from_capacity(100);
    ..
    acc.push_str(&data.upvotes.to_string());
    ..
    acc;
}
pub struct PostData {
    postTitle: String, 
    upvotes: f64
}

fn render_post(post: IPost) -> String {
    let mut acc = String::from_capacity(100);
    ..
    acc.push_str(&data.upvotes.to_string());
    ..
    acc;
}

In terms of doing this, getting the biggest abstraction at a low cost requires a build step to do static analysis on ASTs and do specific code generation.

The other benefit of Prism, Svelte and Vue is that they use single-file components. These allow you to write CSS in the same file as the components.

<template>..</template>
<script>..</script>
<style>
    div.container {
        width: 80%;

        h1 {
            color: red;
        }
    }
</style>
<template>..</template>
<script>..</script>
<style>
    div.container {
        width: 80%;

        h1 {
            color: red;
        }
    }
</style>

There are many benefits in readability in having CSS alongside the components. Prism and others also automatically scope classes so that other `div.container`s are not affected outside of the component

Other Prism changes

Observable Date instances

One design of Prism is to effectively act as if the DOM was a result of a getter and the view should always be 1:1 to the value of the evaluated getter

class MyComponent {
    date = new Date()

    get content() {
        return `<h5>${formatDate(this.data)}</h5>`
    } 

    addMinuteToDate() {
        this.date.setMinute(this.date.getMinute() + 1)
    }
}
class MyComponent {
    date = new Date()

    get content() {
        return `<h5>${formatDate(this.data)}</h5>`
    } 

    addMinuteToDate() {
        this.date.setMinute(this.date.getMinute() + 1)
    }
}

Frameworks have something where you tell it to update with the new state. React's setState is an abstraction over rerendering the DOM as React doesn't have a concept of state. It should be rerenderWithTheseValues. Simply setting a property in a React will not make the view update.

Svelte is better in that its state updates are triggered around the assignment operator. Which is a step towards more "native" JavaScript. However, there are still issues around internal changes. You cannot use push in Svelte, instead x = [...x, newItem] is required for the compiler to realise an update has happened. This is also the case for the Date instance, calling setMonth etc does not cause the view to be updated. With Prism I wanted to allow internal mutation in the same way JS works. So I implemented this for Date.

(just don't look how I implemented this)

Internal updates

In Prism 1.5.0 the Rust SSR compilation was improved so the server render functions now append to the same buffer. The buffer can have initial capacity using the new decorator @WithCapacity(x). Setting the value can improve SSR performance by avoiding reallocations. There are also some unnecessary to_string calls on Strings which have been removed with the help of types.

Text can now be interpolated when alongside other tags. There are fixes for getting data on nullable nodes and there has been A lot of work behind the scenes to allow for Prism components to be compiled on the browser.

Future

Prism is not designed to be the next new framework. Instead, it is an implementation in attempting to experiment on improving the implementations of SSR and hydration in current popular frontend frameworks.

One thing is that it unfortunately has the same name as the syntax highlighting library prism.js which may cause some confusion. When I named the framework, "Prism" was meant to depict the single source that is split into various paths (CSR, SSR, bindings, hydration logic, etc). I wasn't aware of prism.js and its prevalence until shortly after releasing it under that name. It also unintentionally has an extremely similar logo to the database ORM prisma. If interest were to pick up then I may make features more reliable and release it under a new name.

The compiler is a little rough around the edges. It is not intended for production but if you want to try out JIT hydration or Rust compilation you can try the quick start or fork the HN repo.

Footnotes 📜

(1)>
Render to hydrate

For event listeners to be added the runtime needs to find the elements first. Many frontend frameworks do this via "rendering" their tree and comparing the result with the markup. Therefore all the data is hydrated at the start of page load.

(2)>
Types

All values represented in the DOM are strings. innerText is string, getAttribute() returns string. So when getting the value of score from the server-rendered markup it needs to be converted to number instance first. In order to know what instance to convert it to Prism requires to know the type. This is done using a TypeScript generic argument on the class:

// @useRustStatement #[derive(Clone, Debug, serde::Deserialize)]
export interface IStoryItem {
    id: number,
    url?: string,
    by: string,
    score: number,
    @useRustStatement(`#[serde(default)]`)
    descendants: number, // Number of kids
    @useRustStatement(`#[serde(with = "chrono::serde::ts_seconds")]`)
    time: Date,
    @useRustStatement(`#[serde(default)]`)
    kids: Array<number>,
    // This is "type" in the HN API but Rust does not like "type"
    @useRustStatement(`#[serde(rename(deserialize = "type"))]`)
    storyType: "job" | "story" | "comment" | "poll" | "pollopt",
    text?: string,
    title: string   
}
// @useRustStatement #[derive(Clone, Debug, serde::Deserialize)]
export interface IStoryItem {
    id: number,
    url?: string,
    by: string,
    score: number,
    @useRustStatement(`#[serde(default)]`)
    descendants: number, // Number of kids
    @useRustStatement(`#[serde(with = "chrono::serde::ts_seconds")]`)
    time: Date,
    @useRustStatement(`#[serde(default)]`)
    kids: Array<number>,
    // This is "type" in the HN API but Rust does not like "type"
    @useRustStatement(`#[serde(rename(deserialize = "type"))]`)
    storyType: "job" | "story" | "comment" | "poll" | "pollopt",
    text?: string,
    title: string   
}

The types are also used to build Rust `struct` definitions for definite types on their render methods. `@useRustStatement` allows for adding attributes to the struct members.

(3)>
Non reversible expressions

Some expressions cannot be reversed. For example, the date in markup is rendered as a relative string. From "1 day ago" it isn't possible to construct a Date instance of that value as it could be any hour, minute etc of the previous day. There are a possible 86400000 different Date objects that could have been rendered to say "1 day ago". Information has been lost in converting it into its formatted string. So instead an ISO string representation of that Date is added as an attribute on one of the elements so the hydration logic can do return new Date(elem.getAttribute()).

Prism can reverse some expressions e.g. from /i/${id} it produces this expression result.slice(3). See this issue for further details.

(4)>
Streaming

Currently, the response from SSR is buffered and sent and appended as one. This isn't great as the server could start sending the first part of the markup while it is waiting for the data source to respond. The first thing is that Prism doesn't have a way to mark async data (Svelte has await blocks and marko has Await elements) that are probably needed to do static analysis. The second is that fetch response streaming is not yet stable. Most streaming solutions currently use web sockets. Streaming content on initial page load is a feature by the browser but doing it after the page loads for web components is only possible via this slight hack. Also, Rust generators (which would be preferred) for the compiled SSR functions are not yet stable, (although something could be done with closures). Hacker News is a bad example due to not having a direct request to their database and the latency effects from buffering are visible. Streaming is planned for the future but not available in this release.

JIT Hydration

2020-11-24T00:00:00Z

What is hydration? It typically refers to the act of making statically rendered markup interactive.

But what is interactive? Interactivity is added by attaching event listeners and setting the scene for those listeners to be able to change the view. The step aims to make the page indifferent to it being client-side rendered.

So how do frameworks add event listeners to the existing markup?

Many frameworks do this by rendering the markup again. The rendering identifies where the elements with listeners are. And from knowing their position they can attach to existing markup. To render the application first requires hydrating the state. For example with React:

(Something like that I think)

But the issue here is to add the event listeners the component needs to be rendered and for it to be rendered it needs to its data/state.

This is where hydration comes in. To render this "frame" it needs some data.

Standard client-side rendered sites would make a HTTP request to the backend API to get the data on the initial mount. But this is a separate request, there may have been changes to the data in the time between rendering the page, getting it to the client and the client parsing and running the script. The difference in the data could break things. The other issue is that this is a totally new request and (without backend caching) would have to make another request to the database which is expensive.

So the solution a lot of frameworks use is to send the data down as a JSON blob with the request

<head>
    <script>
        window.__data__ = JSON.parse(`
            {
                "articles": [
                    {
                        "title": "Tips for performant TypeScript"
                    }
                ]
            }
        `);
    </script>
</head>
<body>
    ...
    <h1>Tips for performant TypeScript</h1>
    ...
</body>
<head>
    <script>
        window.__data__ = JSON.parse(`
            {
                "articles": [
                    {
                        "title": "Tips for performant TypeScript"
                    }
                ]
            }
        `);
    </script>
</head>
<body>
    ...
    <h1>Tips for performant TypeScript</h1>
    ...
</body>

This has some overhead because the data is effectively sent down twice

There is also the fact JSON is difficult to work with due to its difficulty in representing complex types such as Date and other things like cyclic references etc. Also, some JSON blobs are quite large especially if there are long lists of objects. And using raw JS object literals can be slow due to the way they are parsed.

Additionally, some parts of this object were not used in the render and so will not be needed in the re-render. In a large number of cases, this object contains more data than was needed.

The 'Time to interactive' measurement can be decreased because the whole state has to be parsed evaluated and the application rendered before a single event listener has been added. This problem also occurs for static site generation as well. Any sort of process of generating markup beforehand (which is beneficial) where that markup needs client-side reactivity needs to do some sort of hydration.

This issue is something framework authors and developers are aware of and actively working on:

Partial hydration is one technique that is being considered. Using partial hydration in an app you split up the application so there are multiple sources. Instead of a single hydrate call, there are multiple invocations under some components. This way you can prioritize reactive elements and ignore static elements. See popular RFCs from these frameworks; Svelte, react, angular.

But most implementations seem quite manual to implement. It also seems difficult to how this plays out with what is a static element and what needs to be reactive.

But today I'll introduce a different approach: JIT hydration

The fastest interactivity would be prioritizing adding event listeners. A non VDOM system doesn't need state to find event listeners. Instead, the framework's compiler could mark elements with event listeners and these could be added with the combination of local state in custom elements and this.querySelector("#element_identifier").addEventListener.

The state still needs to be hydrated to be accessible at the JS runtime. But to prevent the double data problem the values would be hydrated from the rendered DOM and they could be done on access just in time (JIT).

This is what I have implemented into my framework Prism. One of the features is JIT hydration:

And this is working quite well here on my Hacker News clone

It is compiled with the Prism compiler and all the rendered data can be JIT hydrated while none of the payloads send any sort of JS(ON) state blobs.

You can read more about the Hacker News clone here

JIT hydration has the effect of a fully hydrated page but at a lower cost to the client.

So what are the features and benefits of Prism's JIT hydration?

On load, the only logic is attaching event listeners. This leads to a decrease in the 'time-to-interactive' (TTI)
Loading state is deferred until needed
Loading in state is done on a per-property basis rather than everything at once so the lazy-ness does not come at a significant cost
Smaller HTML payload sizes due to not needing JSON state
Resolved complications for objects that cannot be represented in JSON (looking at you Date 👀), cycles etc

Although values in the DOM are all strings, Prism converts the strings to the correct types. As can be seen from the id and score values on each story in the state. Values are also cached after being pulled in during hydration, so later accessors don't have to go back to the DOM. The logic for doing this is invariant to the page and so JS is cached between requests. The JSON blob is different and so is sent down on every new full page.

JSON based hydration

<body>
    ...
    <h1 class="title">*post title*</h1>
    ...
    <script id="data" type="application/json">
        {..., "title": "*post title*", ...}
    </script>
    <script>
        const data = 
            document.getElementById('data');
        const state = 
            JSON.parse(data.textContent);
    </script>
</body>
<body>
    ...
    <h1 class="title">*post title*</h1>
    ...
    <script id="data" type="application/json">
        {..., "title": "*post title*", ...}
    </script>
    <script>
        const data = 
            document.getElementById('data');
        const state = 
            JSON.parse(data.textContent);
    </script>
</body>

Data twice in response, parse done after load

With Prism compiler

<body>
    ...
    <h1 class="title" id="p2312">*post title*</h1>
    ...
    <script>
        const state = {
            get title() {
                const element = 
                    document.querySelector("p2312");
                return element.innerText;
            }
        }
    </script>
</body>
<body>
    ...
    <h1 class="title" id="p2312">*post title*</h1>
    ...
    <script>
        const state = {
            get title() {
                const element = 
                    document.querySelector("p2312");
                return element.innerText;
            }
        }
    </script>
</body>

Data loaded from markup and only when requested

Conclusion

While it has its benefits, JIT hydration is difficult to implement. For it to work effectively there needs to be more analysis and code generation work done at build time. It needs to be established where the state is interpolated and how that can be reversed to be read at runtime.

There are also complications as components now have a virtualized or computed state. Another fact is that child components can have the state before the parent does. To do the immediate event listeners addition without state there needs to be markers in the markup as where elements are. Prism does this with a combination of classes added at build time and custom elements. Using these markers alleviates the need to "render" the whole application.

There are also complexities around values that are not represented literally in the DOM. For example, the href on the link tags is interpreted as /i/${id}. Prism has a limited expression reversal and here creates a slice expression to get the id value. But for more complex expressions it breaks down, especially for not 1:1 expressions where it is impossible.

So yeah, this is super early and experimental stuff. If you have any questions or feedback it would be great to leave them in the comments.

You can follow me and the development of Prism @kaleidawave

Hacker News Clone with Prism & Rust

2020-11-10T00:00:00Z

Last week I recreated the Hacker News site. It is built as an isomorphic site with a Prism compiled frontend and a Rust backend. In this post, I will illustrate some of the benefits that using Prism compiler has on this site.

Isomorphic sites with a Rust backend

Currently, most sites are client-side rendered apps built with a framework. client-side frameworks make development easier by using a more declarative programming style. Using the client runtime can add more reactive interactions without going back to the network. client-side frameworks make SPA's easier to develop.

However, a pure client-side site lacks fast start-up times. Before any content is added to the DOM JavaScript must be downloaded, parsed and executed. This relies on the network speed and speed of the client device. There is also the problem that if this process fails then you end up with a white page without any content. Another problem is that each request returns the same empty HTML so there are problems with search engine indexing and link metadata previews between different pages if the bots don't evaluate the JS.

Client-side rendering is acceptable for mostly interactive sites (e.g. Figma) but does not suit more contentful sites where content is more important than reactivity.

So instead of a server do server-side rendering (SSR) to generate the initial page. SSR fixes the above problems as there is no reliance on the client runtime for generating content. SSR responses contain relevant non-unique markup so search engine indexers and link metadata bots can work without them running JS.

This is true for the Hacker News site that renders pages on the server, which is a good architecture as reading posts and comments is more important than interactions.

Server-side rendering client-side code is already being down within frameworks. react-server/dom has renderToString, and Svelte components have a similar method. Then there are isomorphic frameworks like Nextjs and sapper which tie this all in together.

However the above methods require a JavaScript runtime

So in my own framework, I experimented with the feature to compile views to Rust functions. All .prism components are compiled to native Rust string concatenation functions. This has many benefits:

Single source, no need to rewrite for different server languages
Changes to .prism components are reflected and kept inline with the Rust methods
client-side markup and server-side markup are kept in sync preventing issues when making components interactive

It also compiles across the interface definitions to Rust's struct definitions, which retains the strong type safety. The Rust compiler will pick up any other issues with the outputted Rust functions. This is a step above handlebars where type errors are left to runtime.

It is not as complete as Nextjs which builds the whole backend for you. Prism only builds out functions which then must be connected up. But I prefer this design as getting data on the server and the client is very different (not for this example but for applications where you have full control over the stack). Prism is not a full-stack framework as while it exposes the render methods in Rust modules, it doesn't generate the routing needed in Rust.

Prism hackernews uses actix-web under the hood which is in the top 5 fastest backend frameworks in the tech empower benchmarks, of that list raw NodeJS server places 152nd. According to the benchmark it makes actix-web 7x faster than a raw NodeJS server.

For this example it doesn't really make much of a difference as the biggest time here is making HTTP requests to the HN API. A single request to the homepage takes 11 http requests with a lot of them being chained.

Aside from the speed improvements it also means that if you have written your REST API in Rust then you can add SSR to your site reusing the same server and logic rather than having to rewrite server-side data fetching in a new NodeJS server.

It is already possible to server render SPAs in other languages:

Through calling directly to v8 API. However, this has a bit of overhead and isn't as fast as compiling to native. It loses strong typing and can be complex to integrate.
Rewriting the client view code in a templating engine supported in the language. As well as taking a lot of work initially, it is difficult to maintain the client code with changes to the server-side templates. Mismatches can lead to issues when the frontend logic gets a different response than what the client-side was expecting.

JIT hydration

When building isomorphic sites it often makes the frontend more confusing. Now the frontend it is not the start of the world and there is existing markup. The biggest issue from this is recreating the state that was on the server now on the front-end. With both server and client-rendered DOM the view is the same but the JS state is not there on server responses. Getting the state in JS by doing a data fetch is not good as it may be out of sync with the server content. So the approach used by most SSR frameworks is to send down a JSON or JS blob in a script tag that contains the JS object used to generate the markup response.

This is a common practice and can be seen on nytimes where (at the time of this being written) ~75% of the home page response is the state loaded into the window.__preloadedData variable:

There are a few issues with this approach:

Some of the object is not used. Some implementations use Proxys to identify unused "leaves" but this is complex and expensive to do.
Most of the data is already sent down in the markup.
JSON cannot represent types other than string, number, boolean, object and Array. No Date :(

Prism's approach ditches any sort of JSON or JS blob and the server response only sends markup:

<!DOCTYPE html>
<html lang="en">
    <head>
        <script type="module" src="/bundle.1w99.js"></script>
        <link rel="stylesheet" href="/bundle.lxt.css">
        <title>Hacker News</title>
        <meta name="description" content="Prism & Rust based HN clone">
        ...
    </head>
    <body>
        <router-component>
            <main-layout data-ssr>
                <header>
                    ...
                </header>
                <main class="p900">
                    <index-page data-ssr>
                        <ol class="pwf4">
                        <li>
                            <story-preview data-ssr >
                                <div class="buttons">
                                    <button class="p1lb2" disabled>&#9650;</button>
                                    <button class="plwi" disabled>&#9660;</button>
                                </div>
                                <div>
                                    <h2 class="pavi">
                                        <a ... class="p1kxi" href="...">
                                            Fire declared in OVH SBG2 datacentre building
                                        </a>
                                    </h2>
                                    <span>
                                        <span class="pxwh">660<!----> points | by</span>
                                        <a class="p1ztd" href="..."> 
                                            <!---->finniananderson<!----> | 
                                        </a>
                                        <a class="pos3" href="...">
                                            304<!----> comments
                                        </a>
                                    </span>
                                </div>
                            </story-preview>
                        </li>
                        ...
                        </ol>
                    </index-page>
                </main>
                <footer>
                    ...
                </footer>
            </main-layout>
        </router-component>
    </body>
</html>
<!DOCTYPE html>
<html lang="en">
    <head>
        <script type="module" src="/bundle.1w99.js"></script>
        <link rel="stylesheet" href="/bundle.lxt.css">
        <title>Hacker News</title>
        <meta name="description" content="Prism & Rust based HN clone">
        ...
    </head>
    <body>
        <router-component>
            <main-layout data-ssr>
                <header>
                    ...
                </header>
                <main class="p900">
                    <index-page data-ssr>
                        <ol class="pwf4">
                        <li>
                            <story-preview data-ssr >
                                <div class="buttons">
                                    <button class="p1lb2" disabled>&#9650;</button>
                                    <button class="plwi" disabled>&#9660;</button>
                                </div>
                                <div>
                                    <h2 class="pavi">
                                        <a ... class="p1kxi" href="...">
                                            Fire declared in OVH SBG2 datacentre building
                                        </a>
                                    </h2>
                                    <span>
                                        <span class="pxwh">660<!----> points | by</span>
                                        <a class="p1ztd" href="..."> 
                                            <!---->finniananderson<!----> | 
                                        </a>
                                        <a class="pos3" href="...">
                                            304<!----> comments
                                        </a>
                                    </span>
                                </div>
                            </story-preview>
                        </li>
                        ...
                        </ol>
                    </index-page>
                </main>
                <footer>
                    ...
                </footer>
            </main-layout>
        </router-component>
    </body>
</html>

But can still get the state of the application at runtime:

> JSON.stringify(document.querySelector("index-page").data)
> JSON.stringify(document.querySelector("index-page").data)

{
    "stories": [
        {
            "url":"http://travaux.ovh.net/?do=details&id=49471&",
            "title":"Fire declared in OVH SBG2 datacentre building",
            "score":660,
            "by":"finniananderson",
            "id":26407323,
            "descendants":304
        },
        ...
    ]
}
{
    "stories": [
        {
            "url":"http://travaux.ovh.net/?do=details&id=49471&",
            "title":"Fire declared in OVH SBG2 datacentre building",
            "score":660,
            "by":"finniananderson",
            "id":26407323,
            "descendants":304
        },
        ...
    ]
}

So how does it do this?

As Prism is a compiler, it knows the bindings at build time. With the knowledge of bindings, Prism can compile in getters for each property that accesses the DOM. Once values are retrieved from the DOM they are cached in the component state and will act the same way as if the data was instantiated on the client.

The runtime does this in quite a special way to be efficient. Getting the data is only done once that member is evaluated, rather when the document or component loads. If you evaluate temp1.data.stories[2].title it will only load in the 3rd story, and from the story only the title property rather than bringing in every story and constructing the whole state (not that the getting the state from the HTML is expensive, in practice getting the whole state via JSON took 6ms. But in applications with more than 10 stories on the home page the improvement is visible).

Some of the advantages of JIT hydration:

Smaller HTML payload sizes.
Simplicity, all a component needs to have data is SSR DOM content.
Complications for objects that cannot be represented in JSON (DateTime 👀).

Although values in the DOM are all strings, Prism converts the string to the correct types. This can be seen from the id and score value on each story in the state.

Recreating the state from the rendered markup is not a new thing. Many existing SSR sites manually write JS in the runtime script to get values. The actual HN site does this in its vote logic to where it pulls in the story identifer from the elements identifer:

function vote(ev, el, how) {
  var id = el.id.split(/_/)[1];
  var up = $('up_' + id);
  vis(up, how == 'un');
function vote(ev, el, how) {
  var id = el.id.split(/_/)[1];
  var up = $('up_' + id);
  vis(up, how == 'un');

But Prism automates this process for a declarative input. In contrast to the about logic in hn.js, which is handwritten and relies on the fact that the server-rendered element's id includes the id of the post.

Bundle size

The JS bundle for hacker news is 12.6kb uncompressed (3.5kb Gzip or 3.2kb Brotli). This includes:

All the render logic for the pages
All the reactivity
A single-page router
All the client data fetching logic

Actual HN runs on ~4kb of uncompressed JS but then it is not a spa or framework built so I don't think there is any comparison with it. On the other hand, the Svelte HN implementation runs on 30kb of JS (although it doesn't seem to be minified). But when it comes to these small numbers it doesn't matter, as long as it's under 50kb of JS it's pretty responsive.

Coming back to the size, with any abstraction or automated approach there is bound to be more JS outputted than writing it out by hand. Prism automates a lot of stuff, and provides the declarative approach that exists in all other client-side frameworks while keeping the bundle size very low:

<template>
    <div #html="text"></div>
    <span class="by">by {by}</span>
    <ul class="sub-comments" #for="const subComment of subComments">
        <li>
            <This $data="subComment"></This>
        </li>
    </ul>
</template>

<script>
    // @useRustStatement #[derive(Clone, Debug, serde::Deserialize)]
    export interface IComment {
        by: string,
        id: number,
        @useRustStatement(`#[serde(default)]`)
        text: string,
        @useRustStatement(`#[serde(default)]`)
        subComments: Array<IComment>,
        @useRustStatement(`#[serde(default)]`)
        kids: Array<number>
    }

    const maxCommentDepth = 3;

    export class Comment extends Component<IComment> {
        static async getComment(id: number, depth = 1) {
            const endpoint = `https://hacker-news.firebaseio.com/v0/item/${id}.json`;
            const request = await fetch(endpoint);
            const comment = await request.json();
            if (!comment) throw Error(`Could not find comment under id ${id}`);
            if (comment.kids && depth < maxCommentDepth) {
                const subComments = comment.kids
                    .slice(0, 3)
                    .map(kidID => Comment.getComment(kidID, depth + 1));
                comment.subComments = await Promise.all(subComments);
            } else {
                comment.subComments = [];
            }
            return comment;
        }
    }
</script>

<style>
    & {
        display: block;
        margin: 10px 0;
    }

    div {
        font-size: 14px;
    }

    span.by {
        display: block;
        margin-top: 12px;
        font-style: italic;
        font-size: 12px;
    }

    ul.sub-comments {
        padding-left: 24px;
    }
</style>
<template>
    <div #html="text"></div>
    <span class="by">by {by}</span>
    <ul class="sub-comments" #for="const subComment of subComments">
        <li>
            <This $data="subComment"></This>
        </li>
    </ul>
</template>

<script>
    // @useRustStatement #[derive(Clone, Debug, serde::Deserialize)]
    export interface IComment {
        by: string,
        id: number,
        @useRustStatement(`#[serde(default)]`)
        text: string,
        @useRustStatement(`#[serde(default)]`)
        subComments: Array<IComment>,
        @useRustStatement(`#[serde(default)]`)
        kids: Array<number>
    }

    const maxCommentDepth = 3;

    export class Comment extends Component<IComment> {
        static async getComment(id: number, depth = 1) {
            const endpoint = `https://hacker-news.firebaseio.com/v0/item/${id}.json`;
            const request = await fetch(endpoint);
            const comment = await request.json();
            if (!comment) throw Error(`Could not find comment under id ${id}`);
            if (comment.kids && depth < maxCommentDepth) {
                const subComments = comment.kids
                    .slice(0, 3)
                    .map(kidID => Comment.getComment(kidID, depth + 1));
                comment.subComments = await Promise.all(subComments);
            } else {
                comment.subComments = [];
            }
            return comment;
        }
    }
</script>

<style>
    & {
        display: block;
        margin: 10px 0;
    }

    div {
        font-size: 14px;
    }

    span.by {
        display: block;
        margin-top: 12px;
        font-style: italic;
        font-size: 12px;
    }

    ul.sub-comments {
        padding-left: 24px;
    }
</style>

When Prism's HN app launches, the custom elements connected events fire and add event listeners to the existing markup. Here the upvote and downvote buttons on the story preview are connected as event listeners. Prism recognizes that the server is not the client and all elements with events will be server-rendered with the disable attribute (which is removed once the listener is added).

When the button is clicked, the upvote method will fire and inside it will increment this.data.score. If the value is not already in the data cache it will be hydrated in. The increment operator will then fire a set which under the hood fires an imperative DOM call to update the view and the view will now show score + 1 points.

Prism also has a built-in static router. Clicking on the comments anchor tag will route to the story's page. This is just a regular anchor tag under the hood so if the JS fails it will do a regular redirect and hit the server.

SPA routing has the benefits over the MPA that HN has. On a redirect, it only has to load the data for that new view in and not the whole markup. This can reduce the payload. On redirecting the HN header is retained in the DOM and only the page is switched out. Not used here but also animations can be done rather than a flash of content. It also means that this site works without SSR and can be cached in a service worker.

This is only experimental at the moment, there is a lot of stuff hacked together in the compiler to get this example to work. The compilation is shaky at best and only works for simple examples such as this. This is not a drop in replacement for a nextjs site. And lacks a lot of the reactivity that React, Vue and Svelte have to offer.

For now this is a proposal and a experiment to see whether there are improvements that can be made in this.

Future

One improvement that would improve page response times is streaming the document to the client. Currently, the server-rendered page is only sent to the client once the whole server response has been built. Using streaming, the client here would get the header to render in before the backend has even hit the HN API. With each backend request the response could be immediately sent to the client to be rendered in. This would also work really well with the JIT hydration.

With the addition of Rust to the target outputs for Prism it proves that there is the ability to construct SSR functions for other languages. The addition took less than 2 weeks to implement and a lot of that was opening up the pipeline to be less language-oriented (and a lot of wrangling with Rust's memory safety model). There is an opportunity to add support for more languages if this is a beneficial feature but I think the Rust implementation needs to be scrutinized and tightened up before that happens.

This clone is also missing many features from the actual version that I would still like to implement. But I only spun this example up in a week and before then I had no experience with Rust. Also during development Prism was missing a few features such as recursive components (used for the comments) and unsafe html (HN text field includes raw HTML) so I added them for version 1.3.0.

For now, you can clone the clone and try it out here: kaleidawave/hackernews-prism.

For a JS-based isomorphic site, there is kaleidawave/prism-ssr-demo and can be run on replit.

Ben's Engineering Blog

A JSON parser in less than 250 lines-of-code

Usage

Where JSON crops up

Delimited JSON

Building parsers

Gotchas of the approach

Gotchas of parsing JSON

More on performance

Comparing with a full JavaScript parser (Ezno)

Future

Ideas on type inference

Forward inference: pushing a type forwards

The basics of the implementation

Forward inference on properties gives unexpected member warnings for free

Complications

Wrong direction

Backward inference

Steps for backward inference landing in 0.1.0

Allowing valid JavaScript, with new intrinsics

Wrapping up

Specification, speed and (a) schedule

Specification

Speed

A schedule?

Implementing narrowing

What is type narrowing?

Why type narrowing?

Information on types

Narrowing: the implementation

Operator Constructors

Union filtering

Logical operations: appending ands

Logical operations: or

Logical operators: negation operators

Logical operators: De Morgan's laws

Aside: logical operation representation

Control flow

Building new objects

What to do with the narrowed values?

More narrowing

Explicit type annotation type guards

Inference of Not intrinsic

Inference of other number intrinsics

Performance and type overhead concerns

Wrapping up

II lines of code

Base two and base ten

Natural language

Roman numerals

And here is where the neat two line of code snippet I landed on comes in

Try it out

You can download the command line binaries: parse-number and format-number to try it out for yourself.

Testing and benchmarking (Bonus)

Benchmarking

Conclusion

Ezno 2025

Not-so-recent achievements

Adding narrowing

Support for Proxy

Tree-shaking

New lexer design

More on the parser

Immediately next up

Updating the specification

Add a flag to disable the events and partial application system

Finish the inference addition

Add test262 checking

An API for the type system

Future

Continue on improving performance and architecture

More complete-ness: async, Promises, reading .d.ts files etc and focus more on examples

Releasing syntax ideas (under flags)

Other information

Mapped types

The basics of object properties

More advanced shapes of objects

Generic types with generic property keys

Recording generics with the generic chain

Subtyping with generics

Steps for backward inference landing in `0.1.0`

Operator `Constructor`s

Inference of `Not` intrinsic

Support for `Proxy`

More complete-ness: `async`, `Promise`s, reading `.d.ts` files etc and focus more on examples

Passing generics through `key_matches`

Key type modifiers: `in` and `as` syntax

`PropertyValue::ConditionallyExists`

`PropertyValue::Configured`

String intrinsics: `Uppercase`, `Lowercase`, etc

Introducing my own intrinsic: `CaseInsensitive`

Functions and `Object.keys`

Complications with `Object.defineProperty`

Locking objects with `freeze`, `seal` and `preventExtensions`

The current representation for properties: adding `Publicity`

So we want a new kind of type which can represent the negation of some properties or members. This type is called the `Not` type and transforms a single type.

The `Not` type

With our new type at hand we can represent the type of our parameter inside the condition to be `param: string & Not<"">`. (Here `""` is the empty string).

In summary: when left subtyping `Not` then we test disjoint and when disjoint with `Not` we test subtyping.

Aside: `Not` vs `Exclude`

`MultipleOf`

More restrictions with the `Literal` type

Fundamental types: `any` and `never`

Why the intersection with `any` is always `any`