[mstolpm]

Much depends on the topics you cover and the target readers you want to reach. As others have noted, ePub is the standard for ebook publishing. But if you want to publish a printed edition as well in the future, your toolchain would have to support pdf as well.

Pandoc can convert almost everything in everything and is great for terminal nerds. However, be aware that formatting pdfs for common self publishing distributors like KDP (Amazon) has some strict requirements.

For someone without the nerd dns, Vellum is a great software on the Mac that formats ePub and pdf as required for different vendors and print formats. The best feature is that you have one manuscript and export to ePub and pdf. Including graphics works quite well, but sometimes clever positioning for pdf is a bit tricky. Be aware that Vellum is a paid software and it is not suited for equations and docs that require very formal formatting (on the other hand, ePub ebooks are generally not a very good choice here).

But before you choose your toolchain, be aware that writing and formatting the manuscript is only a small part of self publishing. You need distribution and marketing, proof-reading and a business model.

Amazon can help with distribution, but you could publish via your own website, Gumroad and numerous other platforms. Do you want to only sell the ePub? Or sell add-ons, courses, bundles etc.? Amazon is okay for mass market topics, however there is a large competition and you have to have a marketing strategy. Selling via Gumroad allows for bundles and greater flexibility, but is better suited for niche topics. Again: You have to do the marketing.

Does your target group prefer printed books? Amazon/KDP can help here with print on demand. Beware anything where you print (and buy) hundreds or thousands of books in advance: You will have your garage filled with them and never sell them all because of logistics.

So, please don't only focus on the toolchain and the doc to ebook step. That is one of the easier parts in the equation.