refactor: De-couple Chips from a specific ExecutionRecord

[huitseeker]

The Problem

Currently the MachineAir trait requires every chip to define how it interacts with an ExecutionRecord associated type.

This forces any chip that implements MachineAir to only interact with one specific implementation of an ExecutionRecord (as fixed at the moment of choosing that associated type).

We would like those chips to be reused in more varied ways (e.g. with several ExecutionRecords), and the following starts the changes towards accomplishing that.

How it works

We set up the general framework with:

/// A description of the events related to this AIR.
pub trait WithEvents<'a>: Sized {
    /// output of a functional lens from the Record to
    /// refs of those events relative to the AIR.
    type Events: 'a;
}

/// The Record can provide events for the chip T
pub trait EventLens<T: for<'a> WithEvents<'a>> {
    fn events(&self) -> <T as WithEvents>::Events;
}

pub trait MachineAir<F: Field>: BaseAir<F> + for<'a> WithEvents<'a> {
    ...
     fn generate_trace<EL: EventLens<Self>>(&self, input: &EL, output: &mut Self::Record) -> RowMajorMatrix<F>;
    ...
}

(the change to output is similar and pending)

then in AddSubChip:

impl<'a> WithEvents<'a> for AddSubChip {
    type Events = (
        // add events
        &'a [AluEvent],
        // sub events
        &'a [AluEvent],
    );
}

In the ExecutionRecord:

impl EventLens<AddSubChip> for ExecutionRecord {
    fn events(&self) -> <AddSubChip as crate::air::WithEvents>::Events {
        (&self.add_events, &self.sub_events)
    }
}

In generate_trace:

    fn generate_trace<EL: EventLens<Self>>(
        &self,
        input: &EL,
        output: &mut Self::Record,
    ) -> RowMajorMatrix<F> {
        let (add_events, sub_events) = input.events();
        // Generate the rows for the trace.
        let chunk_size = std::cmp::max(
            (add_events.len() + sub_events.len()) / num_cpus::get(),
            1,
        );
        let merged_events = add_events
            .iter()
            .chain(sub_events.iter())
            .collect::<Vec<_>>();
    ...

Important

As a side-effect of this change, each Record defines instantiations of the events it supports precisely, including, in the case of the ExecutionRecord, monomorphically. This means the numerous traits WithAddition, WithDoubling, etc .. we introduced to make a gadget generic over a field / elliptic curve know where to find the right events for itself in the ExecutionRecord are now obsolete and all replaced by this more general pattern: from the PoV of a Chip, a Record is an EventLens<Self> which is thus generally tasked with telling you where to find the events relative to you. (/cc @storojs72 @wwared for the update)

Next steps

• convert the output parameter of generate_trace to not need an ExecutionRecord, then integrate the actual restricted construction from #232,
• convert the prover to not need the ExecutionRecord to be defined as an associated type of the MachineAir(but passed as a "free" parameter),
• same for the recursion prover.

[adr1anh]

This looks really good and is a great first step towards removing the hard-coded dependency on the ExecutionRecord

[wwared]

This PR has many goodies, and I really like it. Especially pleased with all the removed With... traits we had.

The inline comment is just a question for clarifying my own understanding

[wwared]

Asking for my own understanding: so if the Chip is defined as

#[derive(WithEvents)]
enum MyMachine {
    Alu(AluChip),
    Cpu(CpuChip),
}

Then the generated WithEvents trait is something along the lines of

impl<'a> WithEvents<'a> for MyMachine {
    type Events = (
        <AluChip as WithEvents<'a>>::Events,
        <CpuChip as WithEvents<'a>>::Events,
    );
}

And the corresponding EventLens derive looks something like

impl EventLens<MyMachine> for ExecutionRecord {
    fn events(&self) -> <MyMachine as crate::air::WithEvents<'_>>::Events {
        (
            EventLens::<AluChip>::events(self),
            EventLens::<CpuChip>::events(self),
        )
    }
}

(I probably got some minor details wrong above, ignore minor trait/type typos/mismatches)

In other words, every time you call events() on this enum, you're getting the events for all the variants of the chip all at once, correct? But since inside each individual chip's generate_trace function, we call the specialized EventLens method for only that specific chip, we don't pay that cost every time events() is called, only if we call it on the "large" MyMachine enum?

Either way, asking this to highlight one reason that we do not want any manual impl EventLens to ever do any kind of work beyond just returning a reference to a field of some data structure, and why the Events type in impl WithEvents should always be a reference, or this might lead to hidden overhead/unnecessary copies, since the derived Events tuple can end up quite large for some enums

[huitseeker]

That's correct re: how the derive macros work.

Note, however, that you'd have to go out of your way to return anything else than a reference with a WithEvents trait that has a lifetime parameter (thereby telling you "please go and use a reference here").

the derived Events tuple can end up quite large for some enums

See the Proj construction for deducing a smaller EventLens from that large tuple.