I’ve built one of these at FAANG. Generally the different parts of the system are completely separate teams that interact through apis and ingest systems. Usually there’s a mix of online and offline calculations, where features are stored in a nosqldb and some simple model runs in a tomcat server at inference time, or the offline result is just retrieved. Almost everything is precomputed.
We had an api layer where another team runs inference on their model as new user data comes in, then streams it to our api which inboards the data.
On top of this, you have extensive A/B testing systems
I have as well, and your comment matches my experience more than the article does. Different teams own different systems, and there's basically no intersection between "things that require a ton of data/computation" and "things that must be computed online".
Yea same here. What Nosql DB did you use for these lookups? Im currently using postgres for it but seems a bit like a waste. Even though the array field is nice for feature vectors.
The main issue with deploying these systems right now is the technical overhead to develop them out. Existing solutions are either paid and require you to share your valuable data, or open source but either abandoned (rip Crab) or inextensible (most rely on their own DB or postgres).
I’d love to see a lightweight, flexible recommendation system at a low level, specifically the scoring portion. There are a few flexible ones (Apache has one) but none are lightweight and require massive servers (or often clusters). It also can’t be bundled into frontend applications which makes it difficult for privacy-centric, own-your-data applications to compete with paid, we-own-your-data-and-will-exploit-it applications.
It has everything you need at a platform level to build a production recommendation system given that it’s the engine that powered a lot of yahoo product’s search and recommendation capabilities. I have been experimenting with it, the number of capabilities are immense. It’s really an untapped resource.
> As a result, primary databases (e.g. MySQL, Mongo etc.) almost never work
I mean it does. As far as I'm aware Facebook's ad platform is mostly backed by hundreds of thousands of Mysql instances.
But more importantly this post really doesn't describe issues of scale.
Sure it has the stages of recommendation, that might or might not be correct, but it doesn't describe how all of those processes are scheduled, coordinated and communicate.
Stuff at scale is normally a result of tradeoffs, sure you can use a ML model to increase a retention metric by 5% but it costs an extra 350ms to generate and will quadruple the load on the backend during certain events.
What about the message passing, like is that one monolith making the recommendation (cuts down on latency kids!) or micro services, what happens if the message doesn't arrive, do you have a retry? what have you done to stop retry storms?
did you bound your queue properly?
none of this is covered, and my friends, that is 90% of the "architecture at scale" that matters.
Normally stuff at scale is "no clever shit" followed by "fine you can have that clever shit, just document it clearly, oh you've left" which descends into "god this is scary and exotic" finally leading to "lets spend half a billion making a new one with all the same mistakes."
Meta is relatively open (and open source) in how they handle stuff, including ranking, scoring and filtering described in the original article, but also fast inverted indexes and approximate nearest neighbors in high-dimensional spaces. See, for instance, Unicorn [1,2] or (at a lower level) FAISS [3].
> mostly backed by hundreds of thousands of Mysql instances
Kind of. It's part of the recipe but one you find at these large tech companies (I've worked at FB and GOOG) is they have the resources to bend even large/standard projects like MySQL to their will, while ideally preserving the good ideas that made them popular in the first place. There are wrappers/layers/modifications/etc that eventually evolve to subsume the original software, such that is acting more like a library than a standalone service/application. So, for example, while your data might eventually sit in a MySQL table, you'll never know, and likely didn't write anything specific to MySQL (or even SQL) to get there.
Do you think part of this is that Netflix has assumed zero effort from user model? My experience has been that Netflix does an ok job of recommendations, but fails at overall discovery experience. There is no way for me to drive or view content from different angles easily. I end up googling for expert opinions or hitting up rotten tomatoes to get better reviews. Netflix knows a ton about me and their content, but seems to do a poor job of making their content browseable/discoverable overall. I do like their "more like this" feature where I can see similar titles.
In some ways it seems like a classic case of trying to solve the wrong problem because the wrong problem potentially has a technical solution. The real problem is making lots of interesting content for people to watch. If you can solve that problem then a simple system of categories is perfectly sufficient for people to discover content. But that’s not a technical problem, and all those engineers have to be given something to do.
This indicates that the problem is difficult to solve at scale and customized per person. Maybe the issue is with our expectations - I find other people are pretty bad at recommending things for me as well.
Rotten Tomatoes works fine as a recommendation system. It lists all of the new content coming out in a given week. I just read that every week, file down to what looks interesting based on the premise, and read a few reviews. I can usually tell pretty easily what I'll like. No need for in-app recommendations from any specific streaming service at all. Good old-fashioned human expert curators.
I was actually pretty impressed the other day when searching for "shiloh" (which they didn't have) because it showed a bunch of "related" queries to other dog movies (they also didn't have). The available search results were a little lacking though.
Interesting post. On thing to note, this seems to be about "on request" ranking. E.g. googleing something and in 500ms you need the recommended content.
However, a lot of usecases are time insensitive rankings. Like recommending content on netflix, spotify etc. (spotifys discover weekly even has a one week! request time :D).
In which case you can just run your ranking and store the recs in your DB and its much much easier.
This is pretty much what both Netflix and Spotify do. I would argue that there isn't a canonical recommendations stack that FAANG is converging towards, and that's a direct corollary of differing business requirements and organizational structure.
All I know is that Facebook's recommendation systems always show me things that I hate to see. I suppose they may "work" at scale, but at an individual level it's epic failure.
FB needs an Ad-Rev-Share-Model with ALL of its users...
Imagine if FB were to pay a fraction% of how yur data was used and paid you for it...
It may be a small amount, but in super 4th world countries, it could affect change in their lives...
Now imagine that this becomes big... and it works well.
Now imagine that the populous is aware of the hand of god above them just pressing keys to affect land masses (yes I am referring to the game from the 80s)
but this cauterizes them into union building...
So when the people realize their metrics are the product to feed consumerism for capitalistic profits, and decide to organize, what happens?
Is FB going to need a military force to protect their DCs?
---
With "Zuck Bucks" (I still am not sure if true)
This makes this ultimate "company store"
Tokens?
So how get?
How EARN? (What service on FB GENERATES '$ZB'?)
How spend?
WHAT GET? (NFTs?, Goods? Services?)?
The entire fucking model of EVERYTHING FB DOES is to MAP SENTIMENT!
Sentiment is the tie btwn INTENT and SENTIMENTAL VALUE
The idea is to map interest with emotional drivers which make someone buy (spend resources their time and effort went into building up a store-of)...
---
So map out your emotinal response over N topics and forums.. Eval your documented Online comments, NLP the fuck out of that, see what your demos are and build this profile to you....
THEN THEN THEN THEN
Offer an "earnable" (i.e. Grindable by farms and bots alike) -- "Zuck Buck" which is a TOKEN (etymology that fucking word for yourself)
of value...
Meaning, zero INTRINSIC value, Zero accountability (managed by a central Zuck Bank) <-- Yeah fuck that)
And the vaule both determined AND available to you via not INTRINSIC CONTROL, nor VALUE.
> since the user is waiting for the “page” to load, most recommendation requests have a budget of only 500ms or so and it is only possible to score a few hundred items in a request.
This doesn't make much sense to me since a recommendation is rarely needed instantly. Why not spend, say, 10 s constructing a better recommendation while the user is doing something else, during which the recommendation can simply be blank. Obviously if the user requests a recommendation on first visit, you're out of luck, but I'm thinking the typical use case is for a recommendation after the primary reason for visiting has been completed.
Much of the field seems to be fixated on throwing massive compute resources at models with results that can neither be evaluated nor reproduced.
"the Recommender Systems research community is facing a crisis where a significant number of papers present results that contribute little to collective knowledge […] often because the research lacks the […] evaluation to be properly judged and, hence, to provide meaningful contributions"
> a machine learning model is trained that takes in all these dozens of features and spits out a score (details on how such a model is trained to be covered in the next post).
This part was the one I was interested in. As most of the rest are obvious.
How FAANG actually builds their recommendation systems:
Millions of cores of compute, exabyte scale custom data stores. Good recommendations are expensive. If you try to build a similar system on AWS, you will spend a fortune.
Most recommender models just use co-occurrence as a seed, this can actually work pretty well on it’s own. If you want to get fancy then build up a vectorized form of the document with something like an an autoencoder, then use some approximate nearest neighbors to find documents close by. 95% of the compute and storage is just spent on calculating co-occurrence though.
> Millions of cores of compute, exabyte scale custom data stores. Good recommendations are expensive. If you try to build a similar system on AWS, you will spend a fortune.
And then it will be gamed, and become as useless as every other recommendation system already going.
These recommendation systems take control away from individuals over what content they see and replace that choice with black box algorithms that don't explain why you are seeing the content that you are or what other content was excluded. All of the companies who have deployed these content selection algorithms could have also given you manual choice over the content that you see, but chose instead to let the algorithm solely determine the content of your feed, either removing the manual option entirely or burying it so thoroughly that no one bothers to use it.
These algorithms are not benign. They make choices about what information you consume, whose opinions you read, what movies you watch, what products you are exposed to, even which politicians messages you hear.
When people complain about the takeover of algorithms, they don't mean databases or web interfaces. They mean this: content selection or preference algorithms.
We should be deeply suspicious. We should demand greater accountability. We should require that the algorithms explain themselves and offer alternatives. We should implement better. Give control back to the users in meaningful ways
If software engineering is indeed a profession, our professional responsibilities include tempering the damaging effects of content selection algorithms.
Did you know how a news paper used to choose what articles it wanted to run?
Do you know how a TV channel decides to schedule stories?
Humans, its all humans. Looking at the metrics, and steering stuff that feeds that metric.
Content filters are dumb and easy to understand. seriously, open up a fresh account at FB, instagram, twitter or tiktok.
First it'll try and get a list of people you already know. Don't give it that.
Then it'll give you a bunch of super popular but click baity influencers to follow. why? because they are the things that drive attention.
if you follow those defaults, you'll get a view of whats shallow and popular: spam, tits, dicks and money.
If you find a subject leader, for example a independent tool maker, cook, pattern maker, builder, then most of your feed will be full of those subjects, save for about 10% random shit thats there to expand your subject range (mostly tits, dicks, spam or money)
What you'll see is stuff related to what you like and stare at.
And thats the problem, they are dumb mirrors. Thats why you don't let kids play with them. Thats why you don't let people with eating disorders go on them, thats why mental health needs to be more accessible, because some times holding up a mirror to your dark desires is corrosive.
Could filter designers do more? fuck yeah, be we also have to be aware that filters are a great whipping boy for other more powerful things.
Want to dive in to all this stuff but can't find a starting point?
Start with reading my patent!
I was smart enough to see what collaborative filtering (CF) could be early on, and to file a patent that issued. I wasn't smart enough to make it a complicated patent, or to choose the right partners so I could have success with it.
But the patent makes a good way to learn how to get from "what are your desert island 5 favorite music recordings?" over to "here is a list of other music you might like". Basic CF, which is at the core of a lot of this stuff. Enjoy!:
Gentle reminder to anyone reading this that your problems are probably not FAANG problems. If you architect your system trying to solve problems you don't have, you are gonna have a bad time.
"And note that you don’t even have to be at FAANG scale to run into this problem - even if you have a small inventory (say few thousand items) and a few dozen features, you’d still run into this problem. "
Wow, this is something that has been a floater-in-mind for decades ;
I'll top it off with an interview at Twitter with the Eng MGR ~2009-ish?
--
Him: So tell me how you would do things differnetly here at twitter based n your experience?
ME: "Well, I have no idea what your internal processes are, or architecture, or problems, so my previous experience wouldn't be relevant."
I'd go for the best option that suits goals.
[This was my literal response to the question, which I thought was a trap but responded honestly -- as a previous mgr of teams, the "well, we did it at my last company as such"]
Dont reply this way. <--
Here was his statement:
This is a literal quote from a hiring manager for DevOps/Engineering at Twitter:
"Thank god!, We have hired so many people from FB, where that was there only job out of school, and no other experience, and the biggest thing they told me was "well - the way we did this at FB was... X"
--
His biggest concern was engineering-culture-creep...
I'll never understand why people think this is a valid criticism of an article, rather than pointing out an issue they have with the actual content of the article. There's nothing inherently wrong with a company sharing info about the space they operate in. In fact, it should be encouraged as long as what they share is useful.
Off-topic, but how did Netflix manage to get itself inserted into the FAANG acronym anyway? Their impact on the tech industry is trivial compared to all the others. Sure, if you just take out the N it's offensive, but we could have said "GAFA" or "FAAMG" would be more accurate to include Microsoft in their place.
FAANG was created by the TV personality Jim Cramer to talk about high growth tech stocks. At the time Netflix was doubling every year. It was based purely on finance.
It's now been taken over by the tech industry to be shorthand for places that are highly selective in their hiring and tend to work on cutting edge tech at scale.
That being said, the impact of Netflix on tech is pretty big. They pioneered using the cloud to run at massive scale.
The phrase originated w/ Jim Cramer, it refers to the 5 best performing tech stocks(or what were the best performing at the time). Nothing to do with their impact on the field from a technical perspective, just a business perspective.
There was a point in time when FAANG offered the best compensation packages for engineers (Netflix was one of them) - so that's where the term originated from but while it's outdated in many respects (Microsoft is not included, Facebook is now Meta, Google is now Alphabet etc etc) it's still sticky for some reason.
I think the acronym gained prominence before Microsoft's recent 'commitment' to open source. Netflix also seemed to be doing really interesting things scaling out 'disruption' to video delivery at the time. It stuck
FAANG was never about impact on tech industry. Otherwise, MSFT would be part of FAANG. Instead, it's directly related to (1) stock price and (2) compensation.
> "FAAMG" would be more accurate to include Microsoft in their place
In Europe you nearly always see "GNAFAM", which includes Microsoft too. It's certainly weird to exclude MSFT, worth at times more than Amazon+Meta+Netflix combined.
Tangent, but I was recently thinking about how FAANG, is now MAANG, and the definition of mange :
(from a google search, lol)
mange
/mānj/
Learn to pronounce
noun
noun: mange
a skin disease of mammals caused by parasitic mites and occasionally communicable to humans. It typically causes severe itching, hair loss, and the formation of scabs and lesions.
"foxes that get mange die in three or four months"
I find it oddly poetic, but, this is my last day of magic.
greatpostman|3 years ago
We had an api layer where another team runs inference on their model as new user data comes in, then streams it to our api which inboards the data.
On top of this, you have extensive A/B testing systems
splonk|3 years ago
lysecret|3 years ago
priansh|3 years ago
I’d love to see a lightweight, flexible recommendation system at a low level, specifically the scoring portion. There are a few flexible ones (Apache has one) but none are lightweight and require massive servers (or often clusters). It also can’t be bundled into frontend applications which makes it difficult for privacy-centric, own-your-data applications to compete with paid, we-own-your-data-and-will-exploit-it applications.
faizshah|3 years ago
It has everything you need at a platform level to build a production recommendation system given that it’s the engine that powered a lot of yahoo product’s search and recommendation capabilities. I have been experimenting with it, the number of capabilities are immense. It’s really an untapped resource.
Take a look at the features: https://vespa.ai/features and the ranking syntax: https://docs.vespa.ai/en/ranking.html
Really cool stuff, I haven’t even scratched the surface of what it can do.
orasis|3 years ago
KaiserPro|3 years ago
I mean it does. As far as I'm aware Facebook's ad platform is mostly backed by hundreds of thousands of Mysql instances.
But more importantly this post really doesn't describe issues of scale.
Sure it has the stages of recommendation, that might or might not be correct, but it doesn't describe how all of those processes are scheduled, coordinated and communicate.
Stuff at scale is normally a result of tradeoffs, sure you can use a ML model to increase a retention metric by 5% but it costs an extra 350ms to generate and will quadruple the load on the backend during certain events.
What about the message passing, like is that one monolith making the recommendation (cuts down on latency kids!) or micro services, what happens if the message doesn't arrive, do you have a retry? what have you done to stop retry storms?
did you bound your queue properly?
none of this is covered, and my friends, that is 90% of the "architecture at scale" that matters.
Normally stuff at scale is "no clever shit" followed by "fine you can have that clever shit, just document it clearly, oh you've left" which descends into "god this is scary and exotic" finally leading to "lets spend half a billion making a new one with all the same mistakes."
xico|3 years ago
[1] http://people.csail.mit.edu/matei/courses/2015/6.S897/readin...
[2] https://dl.acm.org/doi/pdf/10.1145/3394486.3403305
[3] https://faiss.ai/
judge2020|3 years ago
Same for YouTube itself https://www.mysql.com/customers/view/?id=750 and they use Vitess for horizontal scaling: https://vitess.io/
efsavage|3 years ago
Kind of. It's part of the recipe but one you find at these large tech companies (I've worked at FB and GOOG) is they have the resources to bend even large/standard projects like MySQL to their will, while ideally preserving the good ideas that made them popular in the first place. There are wrappers/layers/modifications/etc that eventually evolve to subsume the original software, such that is acting more like a library than a standalone service/application. So, for example, while your data might eventually sit in a MySQL table, you'll never know, and likely didn't write anything specific to MySQL (or even SQL) to get there.
whimsicalism|3 years ago
unknown|3 years ago
[deleted]
fmakunbound|3 years ago
chuckcode|3 years ago
jeffbee|3 years ago
foldr|3 years ago
edmundsauto|3 years ago
nonameiguess|3 years ago
liveoneggs|3 years ago
voz_|3 years ago
lysecret|3 years ago
However, a lot of usecases are time insensitive rankings. Like recommending content on netflix, spotify etc. (spotifys discover weekly even has a one week! request time :D).
In which case you can just run your ranking and store the recs in your DB and its much much easier.
troiskaer|3 years ago
endisneigh|3 years ago
mrfox321|3 years ago
charcircuit|3 years ago
chudi|3 years ago
siskiyou|3 years ago
samstave|3 years ago
Imagine if FB were to pay a fraction% of how yur data was used and paid you for it...
It may be a small amount, but in super 4th world countries, it could affect change in their lives...
Now imagine that this becomes big... and it works well.
Now imagine that the populous is aware of the hand of god above them just pressing keys to affect land masses (yes I am referring to the game from the 80s)
but this cauterizes them into union building...
So when the people realize their metrics are the product to feed consumerism for capitalistic profits, and decide to organize, what happens?
Is FB going to need a military force to protect their DCs?
---
With "Zuck Bucks" (I still am not sure if true)
This makes this ultimate "company store"
Tokens?
So how get?
How EARN? (What service on FB GENERATES '$ZB'?)
How spend?
WHAT GET? (NFTs?, Goods? Services?)?
The entire fucking model of EVERYTHING FB DOES is to MAP SENTIMENT!
Sentiment is the tie btwn INTENT and SENTIMENTAL VALUE
The idea is to map interest with emotional drivers which make someone buy (spend resources their time and effort went into building up a store-of)...
---
So map out your emotinal response over N topics and forums.. Eval your documented Online comments, NLP the fuck out of that, see what your demos are and build this profile to you....
THEN THEN THEN THEN
Offer an "earnable" (i.e. Grindable by farms and bots alike) -- "Zuck Buck" which is a TOKEN (etymology that fucking word for yourself)
of value...
Meaning, zero INTRINSIC value, Zero accountability (managed by a central Zuck Bank) <-- Yeah fuck that)
And the vaule both determined AND available to you via not INTRINSIC CONTROL, nor VALUE.
---
FB Bots Galore.
hugh-avherald|3 years ago
This doesn't make much sense to me since a recommendation is rarely needed instantly. Why not spend, say, 10 s constructing a better recommendation while the user is doing something else, during which the recommendation can simply be blank. Obviously if the user requests a recommendation on first visit, you're out of luck, but I'm thinking the typical use case is for a recommendation after the primary reason for visiting has been completed.
hallqv|3 years ago
lmc|3 years ago
"the Recommender Systems research community is facing a crisis where a significant number of papers present results that contribute little to collective knowledge […] often because the research lacks the […] evaluation to be properly judged and, hence, to provide meaningful contributions"
https://doi.org/10.1145%2F2532508.2532513
More here... https://en.wikipedia.org/wiki/Recommender_system#Reproducibi...
whiplash451|3 years ago
habibur|3 years ago
This part was the one I was interested in. As most of the rest are obvious.
nikhilgarg28|3 years ago
Good feedback, noted. Will get the next post focused on training within the next couple of days.
arkj|3 years ago
nixpulvis|3 years ago
The devil's in the details, which are surely domain specific and hopefully not too morally questionable.
dinobones|3 years ago
Millions of cores of compute, exabyte scale custom data stores. Good recommendations are expensive. If you try to build a similar system on AWS, you will spend a fortune.
Most recommender models just use co-occurrence as a seed, this can actually work pretty well on it’s own. If you want to get fancy then build up a vectorized form of the document with something like an an autoencoder, then use some approximate nearest neighbors to find documents close by. 95% of the compute and storage is just spent on calculating co-occurrence though.
TheRealDunkirk|3 years ago
And then it will be gamed, and become as useless as every other recommendation system already going.
ocrow|3 years ago
These algorithms are not benign. They make choices about what information you consume, whose opinions you read, what movies you watch, what products you are exposed to, even which politicians messages you hear.
When people complain about the takeover of algorithms, they don't mean databases or web interfaces. They mean this: content selection or preference algorithms.
We should be deeply suspicious. We should demand greater accountability. We should require that the algorithms explain themselves and offer alternatives. We should implement better. Give control back to the users in meaningful ways
If software engineering is indeed a profession, our professional responsibilities include tempering the damaging effects of content selection algorithms.
KaiserPro|3 years ago
Do you know how a TV channel decides to schedule stories?
Humans, its all humans. Looking at the metrics, and steering stuff that feeds that metric.
Content filters are dumb and easy to understand. seriously, open up a fresh account at FB, instagram, twitter or tiktok.
First it'll try and get a list of people you already know. Don't give it that.
Then it'll give you a bunch of super popular but click baity influencers to follow. why? because they are the things that drive attention.
if you follow those defaults, you'll get a view of whats shallow and popular: spam, tits, dicks and money.
If you find a subject leader, for example a independent tool maker, cook, pattern maker, builder, then most of your feed will be full of those subjects, save for about 10% random shit thats there to expand your subject range (mostly tits, dicks, spam or money)
What you'll see is stuff related to what you like and stare at.
And thats the problem, they are dumb mirrors. Thats why you don't let kids play with them. Thats why you don't let people with eating disorders go on them, thats why mental health needs to be more accessible, because some times holding up a mirror to your dark desires is corrosive.
Could filter designers do more? fuck yeah, be we also have to be aware that filters are a great whipping boy for other more powerful things.
BubbleRings|3 years ago
I was smart enough to see what collaborative filtering (CF) could be early on, and to file a patent that issued. I wasn't smart enough to make it a complicated patent, or to choose the right partners so I could have success with it.
But the patent makes a good way to learn how to get from "what are your desert island 5 favorite music recordings?" over to "here is a list of other music you might like". Basic CF, which is at the core of a lot of this stuff. Enjoy!:
https://whiteis.com/whiteis/SE/
nickdothutton|3 years ago
nitinagg|3 years ago
ultra_nick|3 years ago
kixiQu|3 years ago
greesil|3 years ago
ehsankia|3 years ago
MANTAMAN
https://streetsharks.fandom.com/wiki/Mantaman
vikingcaffiene|3 years ago
jeffbee|3 years ago
-TFA
samstave|3 years ago
I'll top it off with an interview at Twitter with the Eng MGR ~2009-ish?
--
Him: So tell me how you would do things differnetly here at twitter based n your experience?
ME: "Well, I have no idea what your internal processes are, or architecture, or problems, so my previous experience wouldn't be relevant."
I'd go for the best option that suits goals.
[This was my literal response to the question, which I thought was a trap but responded honestly -- as a previous mgr of teams, the "well, we did it at my last company as such"]
Dont reply this way. <--
Here was his statement:
This is a literal quote from a hiring manager for DevOps/Engineering at Twitter:
"Thank god!, We have hired so many people from FB, where that was there only job out of school, and no other experience, and the biggest thing they told me was "well - the way we did this at FB was... X"
--
His biggest concern was engineering-culture-creep...
unknown|3 years ago
[deleted]
rexreed|3 years ago
"Fennel AI: Building and deploying real world recommendation systems in production Launched 18 hours ago"
Caveat reader.
warent|3 years ago
imilk|3 years ago
unknown|3 years ago
[deleted]
oofbey|3 years ago
jedberg|3 years ago
It's now been taken over by the tech industry to be shorthand for places that are highly selective in their hiring and tend to work on cutting edge tech at scale.
That being said, the impact of Netflix on tech is pretty big. They pioneered using the cloud to run at massive scale.
dljsjr|3 years ago
vincentmarle|3 years ago
errantmind|3 years ago
yukinon|3 years ago
TacticalCoder|3 years ago
In Europe you nearly always see "GNAFAM", which includes Microsoft too. It's certainly weird to exclude MSFT, worth at times more than Amazon+Meta+Netflix combined.
unknown|3 years ago
[deleted]
cordite|3 years ago
tempest_|3 years ago
unknown|3 years ago
[deleted]
werber|3 years ago
I find it oddly poetic, but, this is my last day of magic.
unknown|3 years ago
[deleted]