This is my project :) Nice to see this here again! If there are questions, I'm ready to answer (I have a rather young baby here though that wanted attention while writing this, but she sleeps now, so I think we are good).
Pipes saw some internal updates recently. It was fighting with some instability. My first measure was an internal re-architeture, on a small scale. Before, the data was transported as text from block to block. (Almost) Every block parsed the input, created an RSS feed object and then worked with that on the data, only to then output the created RSS feed as string. This was changed to just work with the RSS object directly and move that from block to block. That had some consequences for some existing pipes, mainly because it also changed when the input feeds gets normalized.
As that did not help, next step was a server upgrade. I'm not sure whether it was directly the additional processing capacity or the dependency upgrades, but that one helped so far. Knock on wood. Also reconfigured the amount of threads and puma workers, that might have solved a bottleneck somewhere. I sadly never pinpointed exactly what made the server process stop before.
I had played with a proper split between web frontend and pipes processing, but at least my approach was not viable. Worked nicely while developing it, immediately crashed with the production workload. And Pipes is not all that huge... I'll have to try again some time.
What I am missing is a block to manipulate contents. For example, I might want to add an AI generated summary to certain feeds. Or if there is no image, add an alternative image. For those cases it could be useful to have a block that can post a feed item (or one selecteble field) to my custom url and add or replace that item with the result. Then I can make my own feed item manipulators for certain tasks. Thinking bigger, you might offer a (third party?) store of custom manipulators: 'add image', 'add summary', etc.
I see a lot of folks here asking about what happened to the original Yahoo Pipes.
We had the same question, so we went and talked with a lot of the original team and wrote up the story[0]. We also made a fun mini-site that contains a lot of easter eggs (e.g. if you click on the "Memory Pipes" folder on the desktop, you'll see a bunch of candid photos of the original team circa 2007)
By the way, that page is entirely unusable without JavaScript enabled, apparently in service of a modal preloader? Retro vibes are nice, but let's leave Flash-based websites in the past.
Finally! Yahoo Pipes was so ahead of its time. The whole concept of microdata, RSS and wrangling data around in pipes was amazing and it is sad that it died. The web was so promising at that time.
I have no experience with this particular implementation but I do have experience with some important business process developed in Azure Logic Apps, which is kind of the same in spirit I believe.
From what I have seen, these kind of frameworks facilitates the worst possible combination of factors: A solution designed, created and implemented by non-programmers, not using a programming language ending up in a production environment.
One could argue this is a success in its own but I have only seen these kind of things hit their limits almost immediately after the initial POC and evolve into terrible tech debt. Just learn to program or script already.
That being said, I do not want to disrespect the effort into building this and perhaps there is a place for these kind of solutions I yet have to experience.
Fair enough I'd say :) The concern is understandable, but I honestly think Pipes is different.
The operations Pipes can do are at the same time very generic and very targeted. It's stuff like "Download this url" and "Filter this feed for that word", and it gives you a selection of blocks like extract and insert with which you can manipulate the data in the pipe (or the feed, depending how you look at it) a bit more freely. There is surprisingly a lot that can be done with that, especially with a replace block that supports regex, but it's not a deep abstraction layer above things that are not mappable to rather small operations.
At work, we use extensively Palantir Foundry and its [excellent] PipelineBuilder.
I was reluctant at first with this low-code no-code stuff. But after porting custom python code, I must admit it covers 99% of the transformation needs we have.
Plus it keeps a semantic all along, that is much more difficult to understand/maintain by code [especially in Python].
Interesting to see the evolution of Visual programming across many tools. We are working on something similar[0] for low-code/no-code devs focusing on creation of APIs, backend tasks, and AI workflows.
Back in the day I had hundreds of pipes, sub-pipes for all kinds of stuff from porn to price gathering, comparisons, to banking. When Yahoo Pipes died so did my will to port all that to something else.
If I had to do it all again in 2024, what’s a robust self hosted project like this?
every form of visual programming I've ever seen is a singular style of some of a box that does processing and a line that connects that box to another box.
> every form of visual programming I've ever seen is a singular style of some of a box that does processing and a line that connects that box to another box.
Mainly because it's the most straightforward method to:
1) Visualize a contained piece of code / function (box)
2) Show where the output of X goes to (line)
There may be other styles, but this style is the easiest for a newbie to wrap their head around, and is the cleanest method of visualization. The box method can then be used to further wrap a bunch of boxes into a larger box when needed.
Block based visual programming languages don't use lines connecting boxes to each other, which has been widely used for a long time with popular visual programming languages like Snap!, Scratch, Squeak eToys, Blockly, etc.
Also, spreadsheets are an extremely popular form of visual programming that has been around for a long time, which don't use lines to connect boxes either.
many of our users have asked for RSS feeds as output
If your users are asking for it then it's locally popular regardless of how much use it gets on the general internet, and therefore it should be important to you. Local popularity (eg the market for your app) is significantly more important to product design than general popularity.
people totally use RSS feeds still. So many systems just make it by default, and I (and many other people) just use RSS readers for following blogs and whatnot.
It's not a complicated thing to get setup! There's a bunch of details you can flub but at the end of the day it's a big thing of markup. The "good enough" thing is easy.
Is there anyone who doesn't use RSS, whether it is as infrastructure behind the scenes (e.g. for a podcast or some kind of notification system) or explicitly in a feed reader?
Another data point from me: I recently started unsubscribing from as many newsletters as possible, and have organized them into hierarchical tree structures in my RSS feed reader instead, according to priority and topics. With many hundred sources by now, some of them extremely prolific posters, I rarely make it past the top priority ones, but for the rest, there is the occasional browse, and a full-text search that I can use on specific topics.
This setup allows me to consume newsletters and blogposts much more efficiently, and the mailbox gets mainly used for correspondence, not general stuff. Suits me just fine, and as an added bonus: I now more seldom have the feeling my every move-move is monitored and stored by who knows who, who knows where, for who knows what purpose.
I can't judge the economic impact of that for you. But with a service like this, it's just so easy to do that you might want to prioritize it :) If your model is pull based, you create a second endpoint that puts the data you'd normally output directly (otionally minus the JSON dressing) into the content field of RSS items. If it's push based, you could even write into an XML file on your server somewhere and let users access that. In the end, RSS is just some XML dressing around your data, and can be as easily implemented as a html template. Storing your old data (a RSS feed should not contain only one item) might be the biggest hurdle, if you don't do that already.
Or you just send your users to Pipes, they could use a download and "build feed" block to create a feed on their own ;)
Edit: Though I noticed that the endpoint of your loops can be specific actions, like a text message, that wouldn't translate of course. More a "put it into the feed" target action then.
Very cool project. A great demonstration that when you're useful you don't need a fancy interface!
If anyone is interested in a more generic visual-programming language, check out Flyde - https://www.flyde.dev - an open-source visual-programming language I'm working on
tiny bit of feedback : I tried to sign-in with a non gmail address and did not receive the confirmation email. Switched to a gmail address and everything was smooth in the sign-in process.
[+] [-] onli|1 year ago|reply
Pipes saw some internal updates recently. It was fighting with some instability. My first measure was an internal re-architeture, on a small scale. Before, the data was transported as text from block to block. (Almost) Every block parsed the input, created an RSS feed object and then worked with that on the data, only to then output the created RSS feed as string. This was changed to just work with the RSS object directly and move that from block to block. That had some consequences for some existing pipes, mainly because it also changed when the input feeds gets normalized.
As that did not help, next step was a server upgrade. I'm not sure whether it was directly the additional processing capacity or the dependency upgrades, but that one helped so far. Knock on wood. Also reconfigured the amount of threads and puma workers, that might have solved a bottleneck somewhere. I sadly never pinpointed exactly what made the server process stop before.
I had played with a proper split between web frontend and pipes processing, but at least my approach was not viable. Worked nicely while developing it, immediately crashed with the production workload. And Pipes is not all that huge... I'll have to try again some time.
[+] [-] Gys|1 year ago|reply
[+] [-] RyLuke|1 year ago|reply
We had the same question, so we went and talked with a lot of the original team and wrote up the story[0]. We also made a fun mini-site that contains a lot of easter eggs (e.g. if you click on the "Memory Pipes" folder on the desktop, you'll see a bunch of candid photos of the original team circa 2007)
[0] https://retool.com/pipes
[+] [-] jjordan|1 year ago|reply
Everything is so flat and boring now...
[+] [-] paledot|1 year ago|reply
[+] [-] dang|1 year ago|reply
The architecture and software behind Pipes - https://news.ycombinator.com/item?id=22959013 - April 2020 (12 comments)
Show HN: Early-stage Yahoo Pipes spiritual successor - https://news.ycombinator.com/item?id=14440993 - May 2017 (126 comments)
[+] [-] jimbobthrowawy|1 year ago|reply
[+] [-] DonHopkins|1 year ago|reply
[+] [-] theshrike79|1 year ago|reply
And NodeRED is big in the home automation space: https://nodered.org
[+] [-] rapnie|1 year ago|reply
[+] [-] ozarker|1 year ago|reply
[+] [-] klaussilveira|1 year ago|reply
[+] [-] smetj|1 year ago|reply
From what I have seen, these kind of frameworks facilitates the worst possible combination of factors: A solution designed, created and implemented by non-programmers, not using a programming language ending up in a production environment.
One could argue this is a success in its own but I have only seen these kind of things hit their limits almost immediately after the initial POC and evolve into terrible tech debt. Just learn to program or script already.
That being said, I do not want to disrespect the effort into building this and perhaps there is a place for these kind of solutions I yet have to experience.
[+] [-] onli|1 year ago|reply
The operations Pipes can do are at the same time very generic and very targeted. It's stuff like "Download this url" and "Filter this feed for that word", and it gives you a selection of blocks like extract and insert with which you can manipulate the data in the pipe (or the feed, depending how you look at it) a bit more freely. There is surprisingly a lot that can be done with that, especially with a replace block that supports regex, but it's not a deep abstraction layer above things that are not mappable to rather small operations.
[+] [-] blowski|1 year ago|reply
[+] [-] lolive|1 year ago|reply
[+] [-] akshayrajp|1 year ago|reply
[+] [-] TeMPOraL|1 year ago|reply
[+] [-] harinijan|1 year ago|reply
[0] https://buildship.com
[+] [-] anothername12|1 year ago|reply
If I had to do it all again in 2024, what’s a robust self hosted project like this?
[+] [-] Towaway69|1 year ago|reply
[+] [-] ninetyninenine|1 year ago|reply
[+] [-] pacificmint|1 year ago|reply
But after thinking about it, it feels natural, because usually we talk about "Things" and the "Connections" between them.
Could be a companies org chart, a family tree, a network diagram, abstract things like processes. It's all "Things" and "Connections".
Maybe that's why graph theory has so many applications. Because at the end, it's all just graphs with vertices and edges.
[+] [-] high_5|1 year ago|reply
[+] [-] x-complexity|1 year ago|reply
Mainly because it's the most straightforward method to:
1) Visualize a contained piece of code / function (box)
2) Show where the output of X goes to (line)
There may be other styles, but this style is the easiest for a newbie to wrap their head around, and is the cleanest method of visualization. The box method can then be used to further wrap a bunch of boxes into a larger box when needed.
[+] [-] onion2k|1 year ago|reply
[+] [-] DonHopkins|1 year ago|reply
Snap!:
https://snap.berkeley.edu
Scratch:
https://scratch.mit.edu/
Squeak eToys:
http://www.squeakland.org/
Blockly:
https://developers.google.com/blockly
Also, spreadsheets are an extremely popular form of visual programming that has been around for a long time, which don't use lines to connect boxes either.
VisiCalc:
https://en.wikipedia.org/wiki/VisiCalc
Lotus 1-2-3:
https://en.wikipedia.org/wiki/Lotus_1-2-3
Microsoft Excel:
https://en.wikipedia.org/wiki/Microsoft_Excel
Brad Myers' paper answers the age-old argument about whether or not spreadsheets are visual programming languages:
https://news.ycombinator.com/item?id=26061576
[+] [-] wild_egg|1 year ago|reply
Seems like a space that could really use some innovation
[+] [-] micheljansen|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] freeqaz|1 year ago|reply
I need to fix the docs because they are dead: https://web.archive.org/web/20200819221527/https://docs.refi...
Yahoo Pipes was an inspiration though. It was similar to Heroku in many ways.
[+] [-] ReadCarlBarks|1 year ago|reply
[+] [-] jumploops|1 year ago|reply
Do people still use RSS feeds?
I work on a similar product[0], conceptually at least, to Pipes, and many of our users have asked for RSS feeds as output.
Until now I’ve rationalized that as low priority, but maybe I’m wrong?
[0] https://magicloops.dev
[+] [-] onion2k|1 year ago|reply
If your users are asking for it then it's locally popular regardless of how much use it gets on the general internet, and therefore it should be important to you. Local popularity (eg the market for your app) is significantly more important to product design than general popularity.
[+] [-] rtpg|1 year ago|reply
It's not a complicated thing to get setup! There's a bunch of details you can flub but at the end of the day it's a big thing of markup. The "good enough" thing is easy.
[+] [-] InsideOutSanta|1 year ago|reply
[+] [-] stareatgoats|1 year ago|reply
Another data point from me: I recently started unsubscribing from as many newsletters as possible, and have organized them into hierarchical tree structures in my RSS feed reader instead, according to priority and topics. With many hundred sources by now, some of them extremely prolific posters, I rarely make it past the top priority ones, but for the rest, there is the occasional browse, and a full-text search that I can use on specific topics.
This setup allows me to consume newsletters and blogposts much more efficiently, and the mailbox gets mainly used for correspondence, not general stuff. Suits me just fine, and as an added bonus: I now more seldom have the feeling my every move-move is monitored and stored by who knows who, who knows where, for who knows what purpose.
[+] [-] gumby|1 year ago|reply
[+] [-] onli|1 year ago|reply
Or you just send your users to Pipes, they could use a download and "build feed" block to create a feed on their own ;)
Edit: Though I noticed that the endpoint of your loops can be specific actions, like a text message, that wouldn't translate of course. More a "put it into the feed" target action then.
[+] [-] Pufferbo|1 year ago|reply
[+] [-] rcarmo|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] Pawka|1 year ago|reply
Daily. For me RSS feed is the only option to pick what I want to read instead of reading what promoter wants me to read.
If there are other ways - I would be happy to know.
[+] [-] gabigrin|1 year ago|reply
If anyone is interested in a more generic visual-programming language, check out Flyde - https://www.flyde.dev - an open-source visual-programming language I'm working on
[+] [-] BaudouinVH|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] jzemeocala|1 year ago|reply
[+] [-] ozten|1 year ago|reply
My vague memory of how this ended last time was processing cost and lack of a business model.
[+] [-] dbacar|1 year ago|reply