NSA OSS Technologies

[rdtsc]

This caught my eye:

> https://github.com/apache/incubator-pirk

> Employing homomorphic encryption techniques, PIR enables datasets to remain resident in their native locations while giving the ability to query the datasets with sensitive terms.

I can imagine a few scenarios there. One perhaps is when db admin should not find out what someone, possibly working on a classified project is querying.

Or say one compartment / project collected the data and now they want to share it with another project. Those read into the second project don't want to reveal to the first one what they are querying because it would reveal classified information.

Another scenario is a database which has results of possibly illegally intercepted communications. If the NSA can argue that the Constitutionally defined "search" doesn't occur until someone actually performs a search (as in runs an SQL query over the data). Then having PIR capability means being able to break the law but only let as few people as possible do it.

Also https://github.com/redhawksdr is pretty damn impressive. It looks like a complete parallel implementation of GNU Radio. Completed with an IDE and such. Wonder how it compares?

[Spearchucker]

This is pretty common in the commercial world too, and something I've done more than once myself. The obvious use case is storing medical records.

In the UK personal medical records are often stored by systems integrators in datacentres with nebulous locations, and need to be accessed by third parties for things like underwriting life insurance policies.

To protect the data (compliance with the EU data protection act) it's encrypted in transit AND at rest. Access to data by third parties is managed through AMRAs (access medical record authorisation), which are completed by the third party, authorised by the data owner (private individual) and given to the data owner's general/dental practitioner or pharmacist, who is able to access and decrypt and appropriately share the sensitive data.

[jamra]

All valid points, but I think it has more to do with this research being popular in the encryption community. One of the Coursera teachers was working on something like that.

An example of use is video games that help prevent hacking and modification.

I think that there could be some great consumer usage for having the ability to encrypt data, but still be able to search it.

[zitterbewegung]

So looking at the apache incubator project pirk is retired due to inactivity [1]. Is anyone interested in reviving the project? [1] http://incubator.apache.org/projects/index.html#pirk

[secfirstmd]

I guess an example might be that the NSA has access to other partners information (e.g non-five eyes like Germany) and doesn't want their DB folks to be able to see that they are quering against the databases (e.g Merkels phone number)

[Tepix]

Intriguing indeed. Could this tech also be used to query a database or service to plot a route (say for navigating with your car) without revealing the route?

[hueving]

Violating the sanctity of the captive portal license agreement! https://github.com/iadgov/goSecure/blob/master/scripts/wifi_...

:)

[libeclipse]

The captive portal license agreement?

[reiz]

That's nice to see the NSA is contributing to the OSS community. I just randomly picked one of the NSA GitHub repositories, analysed it with VersionEye (https://www.versioneye.com) and found already 25 security vulnerabilities. Who is the best person to contact in this case? Here is the security report: https://www.versioneye.com/user/projects/59479cd06725bd00123....

[0xADADA]

They say it clearly themselves:

> The government benefits from the open source community’s enhancements to the technology.

They're hoping that by putting this code out there, unwitting dupes will then collaborate with them to contribute to the surveillance state.

[wfunction]

Can someone explain how some of projects can be MIT-licensed (or anything-else-licensed) as they claim? Aren't they necessarily in the public domain given that they're works of the U.S. Government?

[dagw]

Certain government agencies and subsidiaries are exempt from having their work considered "government work" and can thus claim copyright if they want. I'm guessing the NSA is such and agency. Also if the work was actually done by a contractor then there are other exemptions.

[lsh]

I've trialed Apache Nifi and it's very powerful. It's also a little unsettling as you use it thinking about how the NSA used it ...

[Toast_]

NiFi looks pretty interesting, what's the learning curve like? Looks like a supercharged yahoo pipes.

[StreamBright]

No wonder Hortonworks wanted to integrate it.

[Laforet]

FWIW, all your crypto is used by NSA as well.

[voltagex_]

https://github.com/ozoneplatform/owf-framework looks very interesting - NSA wrote their own BI tool?

[marcosdumay]

With the selling point being that it tracks you less than other tools. In fact, avoiding tracking seems to be a prevalent line on those projects...

Yes, it's understandable, but still ironic.

[killjoywashere]

OWF came up in a meeting recently -- may not be as awesome as NSA wants you to believe.

[Cryptoholic]

I wonder if all the people who are really suspicious of it in here realize that this (releasing their projects as OSS) has been a thing for a while.

SELinux

Accumulo (a popular NoSQL distributed key-value store)

Apache NiFi (data processing system)

etc.

[microwavecamera]

It's nice to see a push for open security and solutions rather than secretive offensive counter-security. Thanks. :)

[alltakendamned]

One does not exclude the other though

[api]

One of my favorites is the Speck cipher, which has been released before:

https://en.wikipedia.org/wiki/Speck_(cipher)

I'd be very interested in more public cryptanalysis of this. It's a damn simple cipher to implement, and if it were at least as secure as say Salsa20/12 it'd be very nice for all kinds of applications.

[nautilus12]

Does the fact that many of these havent been updated in months or years mean that these are really old projects that effectively hold no value to the NSA and arent close to any of their core operations?

[angry_octet]

For every one of these projects someone spent a considerable effort do the paperwork to get it pushed out, have it signed off as sanitized, non-embarassing, etc. It is a lot of work to get that done in bureaucratic and risk-averse organisations.

If there had been a community contributing back I expect that there would have been more activity, but if it seems like noone will, would you spend your time pushing out regular updates?

[kingbirdy]

Or they've received updates that make them too important to release? But likely it's just that they haven't had any major updates, and NSA internal bugfixes aren't important enough to push out, or aren't given out due to worries that that would be a national security issue somehow

[SomeStupidPoint]

There's a lot of neat things there. (This one looks interesting: https://iadgov.github.io/goSecure/)

Also interesting is splitting the repos: that the NSA and IAD have different repos, and that one seems focused on defensive tech while the other is publishing analysis tools.

I know there's a lot of people who aren't fans of the NSA (or what they do), but I think most of us can see a need for a military-grade organization to research defensive technologies for helping secure our infrastructure. I don't think many of us would be unhappy with the NSA if that's all they did. (Or phrased another way: most of us are unhappy because of how they conduct intel work or compromise defensive capability for offensive ones, eg, that whole business with ECC.)

So I think it's important to respond positively to things like the IAD github page, even if we're not fans in general.

[jamesfe]

I think you're right. It's sad to see many people are looking at these tools and performing a sort of "Allegory of the Cave" by extrapolating, then, the evils that can be done with these tools.

Something, mostly common sense, tells me that we will not find some smoking gun to a crime here in these OSS repos...if anyone wanted that, they can refer to any number of leaks.

Ultimately, I'm happy to see this stuff shared, happy to see others use it and happy to see the OSS community build on it.

[sneak]

"military-grade" doesn't mean anything in the context of crypto. We all use "military-grade" crypto every day. If your argument is that somehow the armed forces are better at computer security than the rest of us, because "military", then I reject it wholeheartedly.

They are criminals and should be disbanded. The US intel community is full of cheats and liars, straight to the top.

http://www.hasjamesclapperbeenindictedyet.com

[alpb]

I wonder why https://www.iad.gov (linked at https://github.com/iadgov) is not using a TLS certificate trusted in normal browsers. I cannot visit the webpage as it uses DoD Root CA, which is not installed on my computer.

[ArneBab]

A danger is that if you use their tech, you might become dependent on their tools.

[blazespin]

I suspect there are a lot of very incredible computer programmers at the NSA and they're probably using just regular open source non security related tools every day. It's good to see that they're contributing back to the OS community what they can.

[sneak]

Let's not forget: these people may be skilled, but they are working against every principle of our community.

https://www-androidauthority-com.cdn.ampproject.org/i/www.an...

[bigon]

Also don't forget SELinux https://github.com/SELinuxProject/ which is a project that comes from the NSA as well

[Cakez0r]

SELinux is on the list :)

[deepnotderp]

I see the PR department is getting smarter...

[Animats]

NSA has an offense side and a defense side, and they're not that tightly coordinated.

[mtgx]

Federal government is required to open source at least 20 percent of its code now:

https://sourcecode.cio.gov/

[miclill_kit]

That was exactly my thought as well.

[headmelted]

Femto (https://github.com/femto-dev/femto) looks pretty interesting to me just based on some work I've needed to do in the past that it would've come in handy for.

It looks like the last commit was over a year ago, though. Is there information I'm not seeing of whether these projects are actively maintained (or still in use at NSA?).

[nautilus12]

Im also curious, without digging into it deeply is it something comparable to Solr or Elasticsearch? If so i wonder how it stacks up

[corpMaverick]

This is nice. It should all be about protecting electronic systems. To help individuals and companies build resilient systems. It protects the USA and the world economy as a whole. It should never be about spying people or even catching criminals IMHO.

[thrillgore]

If they're looking for pull requests, they won't find any salvation in the OSS world.

[aramas]

It's generous to have all in royalty free license.

[pvorb]

Why should it be generous? It's been paid through taxes.

[r3bl]

Fun sidenote: I was the very first civilian to contribute to their GitHub project back in July 2015, when SIMP was the only project they had up on GitHub.

It was literally a one letter change in the README file, but I still have the privilege to call myself the very first civilian to contribute to the NSA's open source project: https://github.com/NationalSecurityAgency/SIMP/pull/1

[jameskegel]

I'd shake your hand