aliguori's comments

Hi, I work at AWS and before that on KVM since it was a thing.

Restricting /dev/kvm these days doesn't make much sense. The interface is designed to be safe for any user. The fact that we started as a character device and not syscalls is just a historical decision.

Kata Containers is a lot of infrastructure for running containers and it uses QEMU to run the actual VMs. Firecracker just replaces the QEMU part and we're eager to work with folks like the Kata community.

I love QEMU, it's an amazing project, but it does a ton and it's very oriented towards running disk images and full operating systems. We wanted to explore something really focused on serverless. So far, I'm really happy with the results and I hope others find it interesting too.

Yes, the root volume on i3.metal is exposed as NVMe and is EBS-backed.

Sorry for that. The timeout behavior on earlier kernels is a bit of a pain.

There's a lot to love about NVMe and timeouts are not actually part of the NVMe specification itself but rather a Linux driver construct. Unfortunately, early versions of the driver used an unsigned char for the timeout value and also have a pretty short timeout for network-based storage.

As mentioned elsewhere in the thread, recent AMIs are configured to avoid this problem out of the box.

Yes, all customer PV instances in EC2 are running in an HVM container and are protected against the guest-to-guest Meltdown vulnerability.

As with all virtual and physical machines, patches are necessary to protect against process-to-process Meltdown within the OS itself. Those are starting to roll out from the respective vendors although it will take time for those to work inside a PV instance.

We're still working the details out upstream but the TL;DR is that the way to address Meltdown with Xen PV is to use nested virtualization so that the outer guest is an HVM or PVH guest.

C5 does not support nested virtualization but i3.metal allows using virtualization technology without nested virtualization.

Both i3.metal and c5 use the same underlying Nitro technology.

I tried to cover this in my re:Invent session and walked through how we have been evolving the Nitro System over the last 5 years. Videos and slides will be posted eventually.

Thanks! Would love to hear more about the counters that your interested in. We've exposed more in C5 than in previous instance types and we are trying to make more available over time in a safe way.

It's exactly the same as with the i3.16xlarge instance type. There are eight 1900 GB drives. In an i3.16xlarge, those eight drives are passed through to the instance with PCIe passthrough but for the i3.metal instance, you avoid going through a hypervisor and IOMMU and have direct access.

There is more coming at re:Invent. We have more talks queued up tomorrow on this too.

You can provision these servers just like any other instance. They work just like any other Amazon EC2 instance (same Nitro System platform as C5).

Disclaimer: I work at AWS on the team responsible for the Nitro System including EC2 Bare Metal Instances.

> Hopefully Amazon will disclose more details.

We will have some more details on how this all works at re:Invent in a couple weeks.

You claim it's objectively bad then only cite subjective things like function, structure, and variable names.

That word doesn't mean what you think it means :-)

Technically that was breaking out of QEMU. It was not KVM specific.

If you break into QEMU, you should be a non-privileged user. If you are using libvirt, you are in a cgroup based jail (basically a container) with SELinux being enforced too. So after breaking into QEMU, you would still need to break out of the container before you could attack anything.

But Nelson's exploit was pretty cool. I initially thought remote code execution wasn't possible and he turned around pretty quickly with the exploit. It's quite impressive.

IBM uses a test called IPAT for hiring. It's more of an IQ test than personality.

Maybe if you work down town but a lot of tech companies are in north Austin where housing is very cheap.

These are all good points and I agree for the most part.

However, I have found that a lot of the benefits of virtualization are often lost because of the flexibility of having a full blown Linux OS. People stick too many services on the host and/or don't know how to properly allocate resources.

I think there is something to a simplified model where you no longer have access to any host environment and can only "fill slots". In practical terms, it makes it easier to implement features, like firmware-style upgrade and rollback of the host, for the masses.

Yes, you can do this with a full blown Linux OS on the host but it requires an operational discipline that most environments don't seem to possess.

I'm not sure I buy the whole "run multiple containers on top of an EC2 instance" thing though. I understand why they're using LXC vs traditional virtualization but it seems to me like it's a case of solving all problems by adding another layer of abstraction.

There are too many different kinds of virtualization so we'll just add another layer of virtualization.

Michael Hines has done a fantastic job getting this series merged. It's a rather invasive change and I am amazed at how quickly it was merged.

I don't think you understand the problem.

The problem is that the PRNG has a weak default entropy source. The same problem existed in the kernel for ages. See http://www.factorable.net

The real advice here ought to be that if you are building an application that generates keys, you should make sure that the system has appropriate entropy before generating the keys. Don't assume the PRNG (whether it's /dev/urandom or OpenSSL) does it for you.