Only fire events that were triggered at future blocks

[knivets]

Fixes #1096 #2310 #3069 #1504.

I'm not sure if this doesn't break any other workflows, but happy to modify the PR to accommodate other use cases. Also if I'm missing something obvious I'm happy to learn how to use this functionality in a way to avoid having to deal with unnecessary old events. I tried raising this issue in one of the linked GitHub issues, but got no reply, that's why I spent some time learning how events are implemented under the hood and came up with a fix.

[knivets]

Following up on this PR. Let me know what would be the best way to move forward. I tried running the testing suite locally but looks like I got rate limited. Thanks

[ricmoo]

I’m still catching up on issues and preparing a minor bump.

Is this problem occurring on an Ethereum network? Or is is another network, like BSC or matic? I worry in general about changes like this, since it shouldn’t be a problem in the first place due to how polling works in v5…

[knivets]

Yes, Ethereum network, but it's a local network instance running via Hardhat.

[ricmoo]

I am wondering if this is a bug in hardhat?

So, you are saying you subscribe to an event on a Hardhat in-memory node, and you get old events? When an event is listened to, it uses getLogs with the current block as the start block. This to me indicates they are returning logs outside the the range of the fromBlock..toBlock. If this is the case, let me know and I'll bring it up with them.

Thanks!

[knivets]

No, the initial getLogs request is performed with fromBlock: -2 parameter, that's why it returns all events.

[ricmoo]

It shouldn’t send a request with a from block of -2, if it is, that is a bug. The -2 is a sentinel value only used for indicating it hasn’t been set yet.

Otherwise most event calls would stall, as large ranges throw…

[knivets]

I ran it through a debugger multiple times, and the value of fromBlock was -2.

[ricmoo]

Ok. I’ll look into that. That would be a bug.

[knivets]

Thanks! Would you like me to create a screen recording where I reproduce the bug?

[ricmoo]

Don’t worry about that too much yet. I’ll add some console.log statements to the code so I can see things and add more context.

[hickscorp]

Hi.
We're following on this discussion and we would like to report some of our findings and how they potentially go against common expectations.

Using an event filter defined using fromBlock: 'latest', we tried both:

• provider.on(...).
• provider.once(() => provider.on(...)).

In both cases, the filter doesn't seem to account for the fromBlock: 'latest' directive, and replays very old events.

This is not what is expected - when listening to events starting at fromBlock: 'latest', the developer's intent is to monitor "things that happen from now on" explicitly.

Is there any way to achieve this behaviour in the current state of things with Ethers? We would like to avoid a "frontend hack" which would still pull historical events from the chain and potentially add some load to it...

Thank you.

[ricmoo]

@hickscorp The .on and .once set up a listener for events and will ignore fromBlock and toBlock. If you want to use getLogs you should either use the provider.getLogs or the contract.queryFilter methods.

That is unrelated to this issue.

[ricmoo]

I have added console.log throughout the BaseProvider and have no been able to see any instances where a fromBlock of -2 is sent as part of the request. Do you have a concise example that can trigger this?

I am using:

(async function() {
    const provider = ethers.getDefaultProvider();
    provider.on({ topics: [ ethers.utils.id("Transfer(address,address,uint256)") ] }, (from, to, amount) => {
        console.log(from, to, amount);
    });
})();

Is there something missing?

[knivets]

Hm, I just tried and it looks like fromBlock is not -2, but the event._lastBlockNumber is -2. At this point I don't remember if it was fromBlock: -2 specifically or event._lastBlockNumber. Here're the params that I have currently:

{topics: Array(1), address: '0x5FbDB2315678afecb367f032d93F642f64180aa3', fromBlock: 8, toBlock: 18}

It would still show me the last 10 events.

[ricmoo]

Eureka!

I think I found the problem.

The range used by the filter is constantly being refined as blocks and events come in. If you are currently on block 18, that range [8, 18] is fine because that is only possible if there were no matching events in blocks [8, 17], because if there was, for example in block 15, that event would have updated the event._lastBlockNumber to 15, so the updated range would be [16, 18]. So, old events should show up, since they weren't there.

That is actually the purpose of this overly complex; the getLogs call returning no events does not necessarily mean there weren't any events in that block, they may exist but not yet have been indexed in the db responsible for responses from getLogs.

So you do not want to restrict to >= blockNumber, otherwise you would skip any event in a recent block that has not had its chance to get emitted. But while typing this response, it gave me an idea to test for...

This "gap logic" is performed on the first poll, which it shouldn't as that is expanding the range to a lower values for the fromValue than it should. I think there should be a fairly simple fix. I'll keep this issue updated. :)

[ricmoo]

As a note, I tracked this down and it was introduced in v5.6.0, which added support for networks which has non-consistent event indexing.

It appears fixed in the above change and will go out in v5.7.0.

Thanks! :)

[knivets]

That's awesome! Great job @ricmoo

[dound]

Is v5.70 going to be released with this? Still on 5.6.9 and looking forward to the fix.

[ricmoo]

@dound yes. I’m still working on it. A little more patience please. :)

[ricmoo]

Yes, sorry. I thought I’d updated all the issues but missed this one. It was published 2 days ago. :)

[dound]

Thanks @ricmoo !! I was a little worried this might have been shifted to v6 release. Thanks for the surprise release today; we've upgraded and I appreciate the patch!

[ricmoo]

@dound no worries. It was too important to push to v6. :)

Let me know if you continue having any issues.