[gnramires]

Good question, and I've spent a few years investigating this sort of question :)

It led me to investigate formalizing ethics and if that would be even possible (so we don't fall into traps like you've mentioned)

I think I've gotten pretty good results which I've sketched here: https://old.reddit.com/r/slatestarcodex/comments/1iv1x1m/the...

More to come. In summary, I became confident that we can, if we're careful, know those things and do something like 'maximize happiness' (as I said, I prefer more general terms than happiness! There's a whole universe of inner experiences beyond just the stereotypical smiley person, I think -- I tend to think of 'maximizing meaning' or 'maximizing well-being').

The basic fact that allows this process to make sense, I guess, is that our inner experiences are real in some sense, and there are tools to study them. Not only are our inner lives real, they make part of the world and its causal structure. We can understand (in principle almost exactly, if we could precisely map our minds and brains) what causes what feelings, what is good and what isn't (by generalizing and philosophically querying/testing/red teaming/etc.), and so on for every facet of our inner worlds.

In fact, this (again in principle) would allow us to make definite progress on what matters, which is our inner lives[1]. I think Susanne Langer put it incredibly well: (on the primacy of experiences as the source of meaning)

"If nothing is felt, nothing matters" (Susanne K. Langer)

This is an experimental fact, we as conscious beings experimentally see this fact is true. So in a way the mind/brain is kind of like a tool which allows us to perceive (with some unreliability and limitations that can be worked with) reality, in particular inner reality.

We can actually understand (with some practical limitations) the world of feelings and what matters. To that we simply experiment, collect evidence and properties about feelings and inner lives, try to build theories that are consistent, robust to philosophical (that is, logical in a high level sense) objections; and then we simply do what is best, or if necessary try out a bunch of ways of life and live out the best way according to our best theories.

---

Addendum: Let's take the benzene ring as an illustrative example of our procedure. Someone claims 'a benzene ring is the happiest thing in the Universe, and we therefore must turn everything into a sea of benzene rings. Destroy everything else.'. Is that claim actually true? Let's explore.

It isn't, I claim: if "Nothing is felt, nothing matters". When you are asleep (and not dreaming or thinking at all, let's suppose) or dead, you don't feel anything. No, thoughts are, and must be, associated with activity in our brain. No information flowing and no brain activity, no thoughts. No thoughts, no inner life. Moreover, thoughts require a neural (and logical) infrastructure to arise. It's logically consistent with how we don't observe ourselves as rocks, gas clouds, mountains, benzene molecules, or anything else: we observe ourselves as mammals with actually large brains. There are immensely more rocks, gas particles, benzene molecules, etc.. then there are mammals in the universe. Yet we experience ourselves as mammals. Benzene molecules, rocks and gas clouds just don't have enough structure to support minds and experience happiness.