vedant_ag's comments

We can do that too!

> 1. Statefulness: simple example, api call that returns a query from the DB that has a high transaction load

We natively handle API and DB calls.

> 2. Non determinism: for example returns a random number, guid or a time

This is a bit tricky. We run the same test multiple times, and ignore the changing fields.

> 3. Privacy requirements for certifications or legislation that mean private data cannot be used in a test environment. > > 4. GDPR laws

We anonymise *any* private data before storing it for replay.

> 5. Authentication

We capture auth headers, and mock auth server responses.

> I presume this is more microservice friendly and monolith unfriendly. But that is probably a reason on the pro side of smaller, bounded services.

Currently yes. APIs can be easily tested. Tests for function calls will require user configuration.

> What about ZeroMQ over UDP? A remote time service. Pulling from a git repo. > > No really, I'm asking because I'm trying to figure out how to deal with all this in my tests right now, and I honestly have no idea. But I doubt Codeparrot is going to be able to intercept and intelligently mock everything.

In theory, *everything* can be recorded and replayed; that's the magic of patching-in at the application layer. *Any* function call can be recorded, and, replayed.

In practice, we support whatever is demanded by our customers. If people are ready to pay for something, we are happy to build support for it.

Thanks Leonid! Your vote of confidence means a lot.

OTel for go requires user code changes. Languages that allow monkey-patching (java, js, python, etc.).

> I wonder if it can be done purely using OpenTelementry (e.g. you depend on typical OTel setup), and then read the data directly from OTel DB.

OTel doesn't work out of the box. OTel usually doesn't collect request or response for any network or db call. 90% of my time is spent on extending the individual agents' code; so that they can collect additional required information, and perform "replay".

> 2. Requests don't happen in a vacuum. They likely have data dependencies on prior requests. I recommend some way of sampling sessions rather than individual requests. Replaying the 3rd request in a series of 6 is likely just going to be exercising failure paths.

This is so true. I ran into this problem when I was trying to implement a kind of sampling that limited total RPS. Thanks for reminding. I am definitely looking into more sophisticated methods of sampling.

> I didn't see any explanation on how results are determined. I think it's important to surface those types of details on the website. I'm not going to watch the video on it in hopes of learning.

Good feedback! :)

Modify requests - yes!

New sessions - What is the use case for it? Its definitely possible.

So true!

> I've built this system at many companies myself.

Interesting! Would I know any of those companies/products?

New API tests can be generated by enabling the agent locally and/or on a staging env.

Its simply a matter of using a baseline of: `production` (usually for regression tests), `staging` or `local`.

- Postman collections have to be created manually by a developer (usually).

- Downstream calls (think DB, 3rd party API calls, kafka) are not handled.

Wow very interesting!

Did you manage to play it back manually and/or via tests? I ran into unexpected challenges while doing this.

> 1. what is the type of service instrumentation needed to capture the data? Wonder why this is needed when typically the data is already captured in an APM log? The instrumentation might add performance and security concerns.

Implementation is very similar to an APM log. So the same performance and security concerns apply. We are working on giving both at the same time (Automated tests, and APM), to reduce overhead.

> 2. what is the sampling logic to capture the traffic? It might compromise the fidelity of the test data and give a false sense of test accuracy.

It is random sampling. I feel, 1M or 10M randomly sampled requests should cover all cases.

> 3. what is the duration of data capture? Is it a week's or month's or quarterly data? Meeting 90% coverage on a week's production sample data will provide a false metric.

I was thinking 1 week should be enough. Maybe we will have to add some custom sampling logic for lesser frequency calls (like monthly crons).

> 4. can it faithfully handle data privacy and customer anonymization? This is critical for API's dealing with PCI and other sensitive data.

Yes. Additionally, for compliance, we offer a self-hosted solution- Our code runs on your servers and no data ever leaves your cloud / on-prem.

I'm running a server in my house. Its a ethereum PoS node. It is oddly satisfying to look at it in the flesh.

It's another pet to me: I have to make sure it has a constant supply of electricity and internet. And requires maintenance from time to time.

Some photos here: https://vedantsopinions.medium.com/eth2-node-at-home-without...

Starting with "I'm just curious", is a good way to make sure I'm asking.

"asking" can often come across as "questioning", especially in workplace chat (like on slack). As a manager I don't want to give this vibe to colleagues (reportees, peers, and seniors).

How often (say what % of PRs) do teams end up doing it?

So teams pick up some refactoring tasks in the sprint (from time to time), and it usually includes code deletion?

This makes sense.