WingNews logo WingNews GitHub [2]
top | item 14440993

Show HN: Early-stage Yahoo Pipes spiritual successor

280 points| onli | 8 years ago |pipes.digital | reply

126 comments

order
[+] zeptomu|8 years ago|reply
I've some experience in visual programming languages (both professional and in my studies) and I advise against the graph-form where you connect nodes and edges.

This does not mean that text is the only way to implement programs (and even if you connect nodes, you're still programming), but maybe a good local optimum is in-between, e.g. interactive shells like Jupyter's Notebook and the Mathematica interface. I know there is LabView, the Blender-Editor and AFAIK some Unreal-Engine tool that uses this model, but bigger programs seem really incomprehensible to me.

It's always easy to show-case 5 programs in these node-&-edge editors, but I do not think it is the best approach for visual languages, as one should not under-estimate the layout problems and how to represent information in boxes.

So, I am all for new ideas in visual programming, but I am not sure if the "free canvas" approach works.

[+] LoSboccacc|8 years ago|reply
I don't think you remember how the yahoo pipes worked. It was a brilliant mix of graphs for flow control, premade boxes for all common data manipulation and scripting for that extra feature if needed

I miss them a lot and appreciate the effort here, but don't forget the visual model for data manipulation wasn't invented here but was testsd refined and proved functional by previous existing services

[+] seanmcdirmid|8 years ago|reply
"Patch and wire" can work in certain contexts, especially for data flow models. You are right they don't scale well (Burnett's scaling up problem), but if the programs are relatively small, they are completely manageable. And it does force you to keep your components relatively small and simple (using compounds to hide wiring for more complex designs).
[+] robinhowlett|8 years ago|reply
I work for SnapLogic (an enterprise visual integration product) - the "free canvas" works great for us with live preview of results/data; here's an example linking a third-party code review tool's webhooks with JIRA: https://i.imgur.com/gNr1gGD.png

All of our integrations can be converted into a API with a click of a button too.

[+] jacquesm|8 years ago|reply
Compared to putting together the net list of even a mildly interesting digital circuit interactively this is actually really easy. Not as easy as doing it with a text editor and some programming language or scripting solution would be but I think that Y! pipes was never aimed at the developers, much more at ordinary users and those are much more comfortable with such visual solutions than they are with text.
[+] shiven|8 years ago|reply
Pipeline Pilot (Biovia/Dassault) - Deeply entrenched, industry-grade visual (+textual(PLSQL)) programming system. Depending on who you ask, it is either the bees-knees or a f*ing cancer.
[+] supergreg|8 years ago|reply
Yahoo Pipes was brilliant. I never saw it as programming but as an advanced feed configuration.
[+] metachris|8 years ago|reply
Interesting thoughts, I would be interested to read more aboute this. Can you recommend any good references to start?
[+] sidegrid|8 years ago|reply
"I've" is not normally used where it is spoken as "I have". This is similar to people using "an" in front of a word starting with "h" where the "h" is pronounced. Not sure why people do this.
[+] beardicus|8 years ago|reply
I'm surprised nobody has mentioned Node-RED yet: http://nodered.org/

Node.js-based with a Pipes-like visual wiring interface. It's quite popular with the Raspberry Pi and Arduino crowd. Lot's of input and output plugins, and you can drop into Node scripting when necessary. I quite like it.

[+] perryprog|8 years ago|reply
Node Red is very cool, especially for IOT type apps.
[+] michaelbuckbee|8 years ago|reply
This is cool, but I feel like Zapier, IFTTT and their open source equivalents have moved far past Yahoo Pipes. How do you see this stacking up?
[+] LoSboccacc|8 years ago|reply
They do a fraction of the data manipulation options available from pipes and they require connectors being custom developed, plus they limit you on result delivery, which cannot be pulled.
[+] kilroy123|8 years ago|reply
Agree Zapier is pretty awesome and pretty powerful.

(I have no affiliation, just a happy customer)

[+] GordonS|8 years ago|reply
Do you know what the most fully featured OSS equivelants are?

I've been trying to find such a system that can read from RSS feeds and post to a Facebook Company Page, but to no avail.

[+] wocram|8 years ago|reply
As far as I Kno, IFTTT only supports simple if a: then b triggers, you cannot have multiple inputs or outputs. Do zapier or others improve on this?
[+] onli|8 years ago|reply
I'm interested in the answer to this questions myself :) For me, this is about the ease of use of the visual programming interface, and the focus on feeds to enable the pipe concept. I do not see this in the same space as Zapier and IFTTT, but I might be wrong!
[+] onli|8 years ago|reply
I'd be interested to know which blocks and functionality is missing to support the use cases you used Yahoo Pipes for! And of course general feedback is always welcome.
[+] insomniacity|8 years ago|reply
Nice. It's not very clear that you need to link the pipeline to the right-hand side. (I can't remember if you had to do that on Yahoo, it's been so long!). It's particularly a problem on a wide screen because the eye starts on the left, and I built my pipeline on the left and it took me a while to figure that out.

But well done on scratching an itch that lots of people have had!

[+] onli|8 years ago|reply
> I can't remember if you had to do that on Yahoo, it's been so long!

I have a screenshot on my disk: https://imgur.com/a/TExPX - it had a bigger pipe output block at the bottom. I can see how the red circle alone is not enough, thanks!

[+] unityByFreedom|8 years ago|reply
Nice. Anyone else here ever use Ab Initio? It's an ETL tool for transferring data between databases and manipulating in the process. Seems very similar.

I've always wanted to recreate something with those ideas because it was a very efficient cross between visualizing code and writing it. This looks pretty similar although for different data sources.

[+] mgkimsal|8 years ago|reply
would like to suggest a couple of "starter pipes" - samples to more easily illustrate some possibilities.
[+] houshuang|8 years ago|reply
This discussion is very interesting to us, as we are building a workflow system for collaborative learning (students might do an activity individually, the output of that activity is aggregated and transformed, students are grouped based on an algorithm that processes data from the previous activity, output from one activity is redistributed to another activity, etc). (quick demo: https://www.youtube.com/watch?v=HQ9AyzLOn3Q)

So this discussion about visual languages, data transformation etc, is very relevant to us. One thing we're working on is how to make data transformation more intuitive... Right now we are using JSONPath to enable selection of one field to aggregate on (ie. you have a form where people input ideas, and another activity that takes a list of ideas, so you can input a JSONPath for the field to get aggregated). However, looking at JMESPath (http://jmespath.org/examples.html), it looks much more powerful. Has anyone seen any examples of graphical interfaces for going from one data representation to another, with preview, selecting fields, aggregation etc?

[+] fareesh|8 years ago|reply
Unrelated to pipes specifically but along the same lines, I feel like app developers ought to conform to a pre-defined standard like schema.org for their respective content so that everything can be inter-operable in theory. That way if I'm using Microsoft Todo or Google Keep or whatever, the potential for Google Assistant or Siri or Cortana to add to whichever Todo App I'm using is already there.

What are the drawbacks behind something like this?

[+] comboy|8 years ago|reply
No incentive. If your service is used through something else then you are easily replaceable and users don't spend time in your app.

edit: Just to be clear, I'd love that too. Hopefully it will start happening when everything is driven more by micropayments rather than ads. I'd imagine you'll have "gui companies" and "service companies".

[+] onli|8 years ago|reply
It's also really hard to do. I mean, imagine how one would achieve interoperability for a project like this. Placing blocks via an API from another service? The language/spec supporting that and everything else would be too complex.

But thankfully we have RSS for the data representation, which is a related idea, just the output side. The core of that idea is what enables this site in the first place.

[+] anonfunction|8 years ago|reply
I used to use yahoo pipes for a few twitter bots I had. There was an rss to twitter webapp that I hooked up and it was fully automated for years before it stopped working.

I didn't see the thin red circle on the right, which I now understand to be the output. I almost gave up before realizing my mistake.

[+] Xeoncross|8 years ago|reply
Using yahoo! pipes I created a lifestream application[0] some years ago that won an honorable mention for the 10k javascript challenge. It was nice to have a dynamic site that required no server work on my part.

Updating it for this would probably be a neat way to test out the RSS feed usage.

[0]:https://github.com/Xeoncross/MicroStream

[+] mortadelegle|8 years ago|reply
Very interesting, I do think that visual programming tends to work better when constrained to a small niche of computing, that's the hypothesis behind https://github.com/AlvarBer/persimmon
[+] onli|8 years ago|reply
Wish I had that when I was still doing machine learning. Looks nice!
[+] Mister_Snuggles|8 years ago|reply
I'd love something like this that I can run locally. From looking at the comments, it looks like Node-RED and NoFlo are two possibilities.

I was intrigued by Yahoo Pipes a while back, but didn't want to invest much in it in case it was shut down. Sadly, that worry was well founded.

[+] jonincanada|8 years ago|reply
I archived all the public yahoo pipes as Python modules -- before it was shut down #oldhardrives
[+] jwives|8 years ago|reply
I'm not sure what the licensing on that would be; would it be possible for you to share that archive?
[+] onli|8 years ago|reply
Wow. Did you do anything with the data? It would be very interesting to see for what it was really used!
[+] hehheh|8 years ago|reply
Neat. Any chance of adding JSON processing where you can pick a key (ala `jq`[0]) and then combine it with RSS or other JSON or what have you?

[0] https://stedolan.github.io/jq/

[+] fiatjaf|8 years ago|reply
I've made something[1] that used "jq for everything" in the past.

RequestHub was initially a way to connect webhooks from one service to API calls to another services, using jq scripting for all the customization.

Later I realized it could be used for processing anything (even text!), not only webhooks, but it was too late. Maybe someday I'll do a Yahoo Pipes revival that will just use jq for scripting inside the boxes.

[1]: http://archive.is/nGyH3, https://github.com/fiatjaf/requesthub.xyz

[+] reubano|8 years ago|reply
If I understand you correctly, you want to be able to join the results of multiple sources? You can do something like that with a library I wrote, riko [1]. A simple example would look something like this:

    from riko.modules import join, fetch, fetchdata

    rss_url = 'http://site.com/rss'
    json_url = 'http://site.com/json'
    json_path = 'path.to.data'

    fetch_conf = {'url': json_url, 'path': json_path}

    rss = fetch.pipe(conf={'url': rss_url})
    json = fetchdata.pipe(conf=fetch_conf)

    joined = join.pipe(rss, other=json)
    next(joined)
You can see the docs for the `join` pipe here [2].

[1] https://github.com/nerevu/riko

[2] https://github.com/nerevu/riko/blob/master/riko/modules/join...

[+] infectoid|8 years ago|reply
Where I work we have a custom made system that was inspired by Yahoo Pipes. We are in the slow process of rewriting it.

Also this reminds me of and IoT solution I was shown recently.

https://nodered.org/

[+] hehejo|8 years ago|reply
What about Talend ETL?

You connect and configure various compontents (Input, Output and Manipulation) to achieve you desired dataflow. It's compiled to Java and really Vera versatile.

[+] Toast_|8 years ago|reply
Do you plan on supporting scraping content via css selectors/xpath/regex?
[+] onli|8 years ago|reply
I'm not sure. There are a couple of existing services that do that already, and which give you a rss feed you could then import here. But if it gets requested I would look whether it is doable in the constraints of this system - at least I remember yahoo pipes supported it, that's a plus.
[+] arjie|8 years ago|reply
Nice. Used to love Yahoo! Pipes What's your data model look like?
[+] onli|8 years ago|reply
Every pipe is a recursive list of blocks, like a tree. Basically, a pipe has a starting block that produces the pipe output. To do that it calls the process function of its input block, which again run their input blocks, ..., the feed blocks serve as tree leaves.