JSONField has been an absolute godsend in combination with Django's ORM. I had been using it with Postgres and will likely keep our backend the same, but I cannot recommend it enough. You will have to write some validation and schema code on top if you want your data to have similar (but weaker) guarantees to the usual typed fields; the benefits from the flexibility you get are immeasurable though.
In projects I've been involved with, storing JSON in the database has often turned out to be a mistake.
Django models create a well-defined self-documenting structure for your schema, are easy to evolve using migrations, and there's a wealth of tooling built on top. IMHO, these far outweigh the perceived convenience of simply storing some stuff in a JSON field.
If you find yourself implementing your own validation and schema code for JSON fields, I'd say it's a sign that you should probably stop and migrate the data to Django models instead.
There are some cases when storing JSON is fine, of course, but in my experience they are few and far between.
We've been using it pretty heavily and the ability to do deep queries like Person.objects.filter(data__family__spouse__name__istartswith="oli") is great for one-offs but the performance stinks.
As it well should —this isn't a complaint— just a reminder that keeping denormalised, first party data that can be well indexed is invaluable when you're talking about huge datasets.
It'd also be nice to layer in JSON Schema directly into the modelling but there is a draft-7 compliant Python project that you can use for this in the model's clean() method.
My intuition says this is one of those tradeoffs that is fast and easy in the short run, but over time becomes technical debt. I haven't used these over the long term, though.
Typical use case:
There are a number of required fields that your app won't work without. Those should be columns, probably non-nullable columns.
Then there're a bunch of optional fields. Traditionally, a number of nullable cols would be created. But they're ultimately messy (need to null check every accesses) and unneeded since you can replace them with a single json item. Keys are always optional, so you always need to check for presence or absence, and modern dbs support json natively in many clauses.
I'm surprised the discussion has derailed into whether or not to use JSON fields... Certainly you don't have to, but they've been in Postgres contrib for a long time.
Async views have dropped, and that is genuinely exciting. It's taken a lot of work, and there's still a ways to go before Django is truly async native, but this is one of the big steps, and being able to do some concurrent IO on a few routes could be a huge saving for some projects that call out.
Otherwise a lot of stuff has to be done in event queues, to avoid blocking the main thread, and sometimes that means a bad UX when users take actions and aren't offered the guarantee that they are complete - in times where that might be the best option, were you not risking blocking thread.
As an FYI: Using a gevent monkey patch has been a way to get async Django for years. Overhead is inefficient in CPU cycles and you need to stay away from doing CPU bound things like Numpy manipulations, but for an app server that’s bound by external API call latency, it practically gives infinite concurrency compared to a thread-per-request model. And no need to worry about event queues. You can feel free to synchronously call requests.get with a 10 sec timeout and still serve prod traffic from a small handful of threads. And most days we don’t even worry about async from a coding perspective. Anyone can email me with questions.
As someone that has an active app still written in Django 1.6. The larger my project and more complicate the more I wanted to ditch the model/view/template separation. And attach methods to the model so that it can be called from everywhere, returning html code in a string directly.
Other times, I wished Django had something analogous to a component, where everything is just encapsulated in a single file. I don't want to separate javascript/html/css view/template. I want a single file I need to update and an easy way to call to render.
The template system is also difficult to use, if you get complicated scenarios.
I needed to show event details in a modal if that was the user preference. But the page could also be embedded.
This lead to me having to render differently depending on the device, whether it was embedded and if it was a popup or not. This lead to an explosion of possibilities, a total 2x2x2 = 8 different views for the same data in the same template.
The most practical way was with if / then statements but that still lead to me repeating html code. And being difficult to reason about and test.
I also got into situation where the template wasn't parsed and updated. Probably because of too many includes or inheritance in the template. For example, I wanted to write a custom css file that would use google fonts that the user selected in the UI. The only way I found to work was to bypass the template and write a function in the view that would render_to_string the css file.
Honestly, for complex ui like this, I prefer to write it as a separate frontend app written in react/typescript. Django template system is great for normal websites (e.g. news/blog/ecommerce websites), but for complex application ui you'll start encountering pains like you mentioned. Implementing complex ui with a lot of interactivity is just easier in react. The drawback is now you have to implement an api layer to bridge django and react, which actually not that bad using django rest framework or even vanilla django views.
I've been using 3.1 alpha/beta for a while and it's got some pretty nice improvements. "JSONField for all databases" is a huge deal IMO, as it simplifies a lot of fairly common use cases.
I've been running uvicorn with gunicorn in production, and it's been solid. Django 3.0 enabled this with ASGI support. It'll still be a while before django is fully async capable, but these are steps in the right direction.
Pardon the rant, but I feel that the advantages don't outweigh the fact that with each release some of my stuff gets broken and I need to adjust. Django puts DeprecationWarnings basically everywhere and it's a hell to maintain projects that had been alive for a few years. God forbid you do anything with the interfaces they expose. The problem is only getting worse when you consider your dependencies, which in many cases don't keep up with the release cycle of Django. It's a mess.
My experience has been good, with a Django app maintained for over 8 years now. We've only updated Django when a new LTS version is out. That means once every two years there's an update that may break something, but I don't remember it ever being more than a few hours of work.
We do take some care to (mostly) only use documented interfaces, so maybe that helps?
I'm sorry, but this is just programming. You have two choices: 1. Never upgrade and keep everything stable but go through hell when you inevitably must update something or 2: keep up with the latest and go through some minor pain with every release.
It's a lot like taking care of a house or car.
The world does not stand still, much less programming frameworks.
I tend to keep one CI pipeline with all dependencies unpinned (I made pip-chill for that reason) and Django set to the latest version to get early warnings of the impending doom.
That said, it takes a while for the Django folk to end support for a release so, at least, there is not much of a hurry to update a running app.
This seems like exactly the way that deprecations should be handled. Are you suggesting they should maintain backwards compatibility for much longer? They already have a pretty long deprecation policy.
Can you elaborate on some of the things you've been frustrated with?
I find Django has a pretty solid deprecation policy, with the goal being that if you have no deprecation warnings then upgrading is simple. Third party dependencies do get in the way of that sometimes though.
I'm only a beginner with Django, but I tend to agree to a point - reason being is that I've found old code that I want to use (github, etc.), but it targets Django 1.x (later versions, say 1.10), and will use a load of stuff which is now removed. The issue I've had is that the documentation seems to be difficult to find these removed elements - such as last week, looking for the {% ssi (file) %} tag, which it took me the best part of an hour searching to find out which version it came from - and indeed that it was a standard tag!
A lot of the imports seem to move around (which is easy to fix, once you know where they now are), and some things have just vanished.
I know documentation is a nightmare to produce, but there must be a way to automatically produce it for either import relocations or deprecations so you can find things like this easily?
For me, upgrading Django for a series-A sized startup (e.g. 25KLOC) takes maybe 2 - 4 hours on average, assuming there is good integration test coverage. What parts of the upgrade process are you finding take a long time?
I feel like the last Django release that was at all complicated to upgrade to was maybe 1.9, in terms of getting the test cases to run properly in parallel if they hadn't been properly isolated. And even that was more of a Python3-like situation, where it really only exposed things that people had done incorrectly previously.
Compared to how fast things move in frontend world, Django feel like ultra stable to me. Upgrading an old django project to the latest version typically doesn't affect any fundamental things. Maybe some classes has been moved into contrib or spin off as non-core packages, or some methods has been removed and replaced with another methods, but never fundamental changes that requires you to rewrite majority of your code.
How does HN feel about the recent craze to make web python asynchronous?
To me, the performance gains are dubious in many cases and the complexity overhead of handling cooperative multitasking just seems like a step back for a language like python.
Long term Python user: if I want to do something asynchronously I reach for Go or Elixir (unless it's just way more practical to do it right there in Python). Adding function colors [1] might have been a practical decision, but was IMHO a mistake.
Why do I have to decide if my function is synchronous or not when I write it? I don't want to do that, I want to write only one function that can be called synchronously or asynchronously. In Go or Elixir, the decision is made by the caller, not by the API designer.
Which leads me to a parallel universe: Go-like asynchronicity should have been introduced with Python 3, when backward compatibility was explicitely broken. The gain of switching to Python 3 would then also have been a much easier sell than "just" fixing strings.
Of course, there are probably a thousand things that I'm overlooking, but this is my feeling...
99% of Django apps are CRUD apps with zero need for this. It's easy to get sucked into new-hammer-ism where you have a new hammer and start seeing nails where there aren't any.
The 1% where this is needed does exist, but I suspect that there are far more people using the new async features than actually have need for them. And if you don't need them, you're introducing a lot of complexity, without mature tooling around it to reduce that complexity.
Probably 5 years from now there will be mature tooling around this stuff that lowers the complexity so that it is a good tradeoff for average websites. But for now, I don't need to be an early adopter.
Well, if you have external API calls in your Django app and you are running sync (which I would absolutely advice, with running async it is really easy to get an unpredictable performance which is sometimes hard to track down) having the ability to run some views async is really crucial.
Otherwise your application might me humming along smoothly at some point and coming to a sudden complete standstill or performance plummets when a random external API endpoint starts to time out. Yes I have been bitten by this :-)
To fix this while running sync I have dedicated separate application processes for the views that do external calls, but this makes the routing complex. Alternatively you can juggle timeouts on the external API calls but this is hard to get right and you need to constantly keep track if calls are not timed out just because the external endpoint is a bit slower at some point.
So I think this solves a very real-world challenge.
90% of the time I don't want it. Database, cache, etc, not really that bothered. Web requests take 100-300ms to complete, tying up a worker for 300ms isn't much of a problem.
10% of the time I'm calling an API that takes 3s and tying up a worker for 3s _might_ be a problem. Being able to not do that would be really handy sometimes.
Not web servers, but I also do a lot of web scraping and Python is definitely the best tool I've used for that job (lxml is fast with great XPath support, Python is very expressive for data manipulation), using async for that could dramatically improve our throughput as it's essentially network bound, and we don't really care about latency that's in the 10s of seconds.
Five times later, there are some new frameworks, but much of the ecosystem is still sync-only. This is actually one of the things that is pushing me towards Go lately. Python just doesn't seem to mature fast enough and tools heavily disagree on conventions.
If I look at the database queries on the vast majority of pages in a typical Django project, I see a big list of operations being executed sequentially that could actually be done in parallel.
Additionally, (this is my pet use case) if you implement a GraphQL server on top of Django (using one of the many libraries), you tend to get subpar performance because GraphQL lends itself really well to parallelised data fetching, which is hard to take advantage of at the moment.
Adding new SQL support to the Django ORM is tricky. There isn’t a nice low level abstraction for generating and composing SQL. It’s mostly a bunch of lists containing different kinds of data (strings, Col, Expression) and they don’t compose well.
On top of that, you’d need to come up with a decent frontend syntax that aligned with the existing methods.
I think Django made a mistake when first defining the language of aggregates by overloading the values() method and not using group(). To support rollup, values() would need to support it but only when used after an annotate. Not nice.
I often think about what it’d take to use alchemy for the sql building beneath the Django frontend. That would open up so many more possibilities and features.
I work in a C# shop that has added Python to our development, because it runs a long side the poweshell scripts our operations guys build in the managed azure services.
I think Django is just a good package. It’s really productive and the ORM is better and easier to use than entity. The real winner for me is it’s stability though. In the time we’ve gone from classic ASP to web-forms to MVC to modern MVC to .net Core and now soon the new .Net version. Django has remained relatively stable and relatively easy to upgrade.
And make no mistake, we absolutely still operate web-forms applications that will likely never get rewritten in something else.
At the end of the day, tech stacks aren’t really that important though.
The Django Admin is Django's "killer app." If a significant portion of your application is a back-office CRUD admin interface, Django is perhaps the most productive way to build something like that without writing a bunch of forms or using something even more out-of-the-box {like a Drupal). If you become proficient with Django you can build custom CRUD admins very quickly.
For other use cases, there are better frameworks out there.
PHP on its own was more of a templating system plus arbitrary code when I used it. Django and ASP.NET have ORMs and templating systems.
Lots of people love ORMs although I’ve found complex queries slow with a hefty object model system, to the point where I’d rather write parameterised sql queries than work with a ORM optimization strategy.
The templating system is nice although these days I mostly use javascript talking to json endpoints, with nearly zero need for a templating system.
Honestly when your need is: “javascript talks to endpoint and endpoint talks to database” I don’t see a greater need than python, nodejs, golang or whatever language you prefer plus a couple of libraries. Most server frameworks add more stuff you probably don’t need, unless you can’t work without an integrated ORM.
We actually had to make that choice 10 years ago. .NET was out because we didn’t want to deal with Windows server, but that’s no longer an issue. Now I think I would skip ASP.NET because everything is confusing as all hell. You don’t know which .NET supports which feature, or even what you’re suppose to be running. It’s the fastest of the three and Visual Studio is still an awesome IDE.
We dropped PHP due to the lack of any really good frameworks, but now we have Laravel. PHP is still a solid choice and it’s fast.
In the end we went with Django because we liked Python and Django is really well documented and easy to learn.
Something I wish that Django did was user defined functions in the template. It has for loops, which is good but it forces you to write the html in top to bottom procedural manner.
It would be far better to be able to define a function that you can call for bits of html code that might repeat in the same template.
Since I stopped using Django at 1.6 does the new version let you define functions in the templates?
I'm closer to a hobbyist than a professional dev, but the async views seem like a big functionality. Having done some Django apps, getting a synced up view for some changing variable was always a bit painful.
async is cool for I/O bound operations. You don't have to wait the request/response to finish in order to start processing another request.
Talking with a db, or doing http requests are IO operations, so instead of blocking the process, django can now start processing another one. When the IO operation is done, it continues where it stopped.
it would be useful if your code has to execute async code and await it's result from within a view. For example: create 10 tasks and return await asyncio.gather(*tasks) as json.
grenoire|5 years ago
metafunctor|5 years ago
Django models create a well-defined self-documenting structure for your schema, are easy to evolve using migrations, and there's a wealth of tooling built on top. IMHO, these far outweigh the perceived convenience of simply storing some stuff in a JSON field.
If you find yourself implementing your own validation and schema code for JSON fields, I'd say it's a sign that you should probably stop and migrate the data to Django models instead.
There are some cases when storing JSON is fine, of course, but in my experience they are few and far between.
oliwarner|5 years ago
As it well should —this isn't a complaint— just a reminder that keeping denormalised, first party data that can be well indexed is invaluable when you're talking about huge datasets.
It'd also be nice to layer in JSON Schema directly into the modelling but there is a draft-7 compliant Python project that you can use for this in the model's clean() method.
kerkeslager|5 years ago
My intuition says this is one of those tradeoffs that is fast and easy in the short run, but over time becomes technical debt. I haven't used these over the long term, though.
robertlagrant|5 years ago
nojs|5 years ago
sandGorgon|5 years ago
alanfranz|5 years ago
Then there're a bunch of optional fields. Traditionally, a number of nullable cols would be created. But they're ultimately messy (need to null check every accesses) and unneeded since you can replace them with a single json item. Keys are always optional, so you always need to check for presence or absence, and modern dbs support json natively in many clauses.
sammorrowdrums|5 years ago
Async views have dropped, and that is genuinely exciting. It's taken a lot of work, and there's still a ways to go before Django is truly async native, but this is one of the big steps, and being able to do some concurrent IO on a few routes could be a huge saving for some projects that call out.
Otherwise a lot of stuff has to be done in event queues, to avoid blocking the main thread, and sometimes that means a bad UX when users take actions and aren't offered the guarantee that they are complete - in times where that might be the best option, were you not risking blocking thread.
btown|5 years ago
no_wizard|5 years ago
jungletime|5 years ago
Other times, I wished Django had something analogous to a component, where everything is just encapsulated in a single file. I don't want to separate javascript/html/css view/template. I want a single file I need to update and an easy way to call to render.
The template system is also difficult to use, if you get complicated scenarios.
I needed to show event details in a modal if that was the user preference. But the page could also be embedded. This lead to me having to render differently depending on the device, whether it was embedded and if it was a popup or not. This lead to an explosion of possibilities, a total 2x2x2 = 8 different views for the same data in the same template.
The most practical way was with if / then statements but that still lead to me repeating html code. And being difficult to reason about and test.
I also got into situation where the template wasn't parsed and updated. Probably because of too many includes or inheritance in the template. For example, I wanted to write a custom css file that would use google fonts that the user selected in the UI. The only way I found to work was to bypass the template and write a function in the view that would render_to_string the css file.
mixmastamyk|5 years ago
This was always possible.
neurostimulant|5 years ago
scrollaway|5 years ago
Jonnax|5 years ago
ASGI has kinda passed me by. For some small project's I use Gunicorn.
What's the setup for ASGI that's popular?
iaml|5 years ago
I think in python world uvicorn is more popular choice, might be wrong tho
laactech|5 years ago
neillyons|5 years ago
VWWHFSfQ|5 years ago
simonw|5 years ago
d33|5 years ago
metafunctor|5 years ago
We do take some care to (mostly) only use documented interfaces, so maybe that helps?
WD-42|5 years ago
It's a lot like taking care of a house or car.
The world does not stand still, much less programming frameworks.
rbanffy|5 years ago
That said, it takes a while for the Django folk to end support for a release so, at least, there is not much of a hurry to update a running app.
danpalmer|5 years ago
This seems like exactly the way that deprecations should be handled. Are you suggesting they should maintain backwards compatibility for much longer? They already have a pretty long deprecation policy.
orf|5 years ago
I find Django has a pretty solid deprecation policy, with the goal being that if you have no deprecation warnings then upgrading is simple. Third party dependencies do get in the way of that sometimes though.
djaychela|5 years ago
A lot of the imports seem to move around (which is easy to fix, once you know where they now are), and some things have just vanished.
I know documentation is a nightmare to produce, but there must be a way to automatically produce it for either import relocations or deprecations so you can find things like this easily?
Alex3917|5 years ago
I feel like the last Django release that was at all complicated to upgrade to was maybe 1.9, in terms of getting the test cases to run properly in parallel if they hadn't been properly isolated. And even that was more of a Python3-like situation, where it really only exposed things that people had done incorrectly previously.
neurostimulant|5 years ago
konschubert|5 years ago
How does HN feel about the recent craze to make web python asynchronous?
To me, the performance gains are dubious in many cases and the complexity overhead of handling cooperative multitasking just seems like a step back for a language like python.
Xixi|5 years ago
Why do I have to decide if my function is synchronous or not when I write it? I don't want to do that, I want to write only one function that can be called synchronously or asynchronously. In Go or Elixir, the decision is made by the caller, not by the API designer.
Which leads me to a parallel universe: Go-like asynchronicity should have been introduced with Python 3, when backward compatibility was explicitely broken. The gain of switching to Python 3 would then also have been a much easier sell than "just" fixing strings.
Of course, there are probably a thousand things that I'm overlooking, but this is my feeling...
[1] https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...
kerkeslager|5 years ago
The 1% where this is needed does exist, but I suspect that there are far more people using the new async features than actually have need for them. And if you don't need them, you're introducing a lot of complexity, without mature tooling around it to reduce that complexity.
Probably 5 years from now there will be mature tooling around this stuff that lowers the complexity so that it is a good tradeoff for average websites. But for now, I don't need to be an early adopter.
whouweling|5 years ago
Otherwise your application might me humming along smoothly at some point and coming to a sudden complete standstill or performance plummets when a random external API endpoint starts to time out. Yes I have been bitten by this :-)
To fix this while running sync I have dedicated separate application processes for the views that do external calls, but this makes the routing complex. Alternatively you can juggle timeouts on the external API calls but this is hard to get right and you need to constantly keep track if calls are not timed out just because the external endpoint is a bit slower at some point.
So I think this solves a very real-world challenge.
danpalmer|5 years ago
10% of the time I'm calling an API that takes 3s and tying up a worker for 3s _might_ be a problem. Being able to not do that would be really handy sometimes.
Not web servers, but I also do a lot of web scraping and Python is definitely the best tool I've used for that job (lxml is fast with great XPath support, Python is very expressive for data manipulation), using async for that could dramatically improve our throughput as it's essentially network bound, and we don't really care about latency that's in the 10s of seconds.
Source: I work on a large production Django site.
maxmalysh|5 years ago
Django provides `sync_to_async` and `async_to_sync`, but it's trivial to do this yourself without Django:
https://docs.djangoproject.com/en/3.0/topics/async/#async-ad...
You can write sync code and use async calls only when needed.
Also, async python is awesome. Things were messy 2-3 years ago, but everything is so much better now.
martpie|5 years ago
This should have no (big) performance impact, but these will most probably allow better concurrency, which can be quite critical for a web framework.
d33|5 years ago
https://news.ycombinator.com/item?id=23218782
Five times later, there are some new frameworks, but much of the ecosystem is still sync-only. This is actually one of the things that is pushing me towards Go lately. Python just doesn't seem to mature fast enough and tools heavily disagree on conventions.
czam|5 years ago
HN submission: https://news.ycombinator.com/item?id=24048208
andrewingram|5 years ago
Additionally, (this is my pet use case) if you implement a GraphQL server on top of Django (using one of the many libraries), you tend to get subpar performance because GraphQL lends itself really well to parallelised data fetching, which is hard to take advantage of at the moment.
est|5 years ago
leetrout|5 years ago
I’d never used it before and it was fantastic but I had to drop down to raw sql to do it. SQLAlchemy has had support for well beyond a year.
I’ve used Django since 2008 and I love it with all its warts but I’ve really grown to prefer SQLAlchemy.
jsmeaton|5 years ago
On top of that, you’d need to come up with a decent frontend syntax that aligned with the existing methods.
I think Django made a mistake when first defining the language of aggregates by overloading the values() method and not using group(). To support rollup, values() would need to support it but only when used after an annotate. Not nice.
I often think about what it’d take to use alchemy for the sql building beneath the Django frontend. That would open up so many more possibilities and features.
madelyn|5 years ago
kumarvvr|5 years ago
If so, why?
moksly|5 years ago
I think Django is just a good package. It’s really productive and the ORM is better and easier to use than entity. The real winner for me is it’s stability though. In the time we’ve gone from classic ASP to web-forms to MVC to modern MVC to .net Core and now soon the new .Net version. Django has remained relatively stable and relatively easy to upgrade.
And make no mistake, we absolutely still operate web-forms applications that will likely never get rewritten in something else.
At the end of the day, tech stacks aren’t really that important though.
pyrophane|5 years ago
For other use cases, there are better frameworks out there.
tarkin2|5 years ago
Lots of people love ORMs although I’ve found complex queries slow with a hefty object model system, to the point where I’d rather write parameterised sql queries than work with a ORM optimization strategy.
The templating system is nice although these days I mostly use javascript talking to json endpoints, with nearly zero need for a templating system.
Honestly when your need is: “javascript talks to endpoint and endpoint talks to database” I don’t see a greater need than python, nodejs, golang or whatever language you prefer plus a couple of libraries. Most server frameworks add more stuff you probably don’t need, unless you can’t work without an integrated ORM.
mrweasel|5 years ago
We dropped PHP due to the lack of any really good frameworks, but now we have Laravel. PHP is still a solid choice and it’s fast.
In the end we went with Django because we liked Python and Django is really well documented and easy to learn.
ttsda|5 years ago
unknown|5 years ago
[deleted]
justinmeiners|5 years ago
For simpler sites Laravel vs Django probably just comes down to which ecosystem you are most familiar with.
unknown|5 years ago
[deleted]
rs999gti|5 years ago
We are a .NET shop, we'd love to use Django, Drupal, etc. But does that also mean we need a dedicated Python resource to support these CMS'?
Or could we use these CMS' out of the box?
Or would we just be better off with a .NET CMS like Umbraco or Orchid?
unknown|5 years ago
[deleted]
anentropic|5 years ago
favadi|5 years ago
jungletime|5 years ago
It would be far better to be able to define a function that you can call for bits of html code that might repeat in the same template.
Since I stopped using Django at 1.6 does the new version let you define functions in the templates?
__alexs|5 years ago
madelyn|5 years ago
It's called "dashml" on PyPi if anyone is interested. I mostly use it in Django and haven't worked on it lately cause of $DAYJOB though.
drcongo|5 years ago
jtdev|5 years ago
stefano|5 years ago
waihtis|5 years ago
StavrosK|5 years ago
FreeHugs|5 years ago
woile|5 years ago
Talking with a db, or doing http requests are IO operations, so instead of blocking the process, django can now start processing another one. When the IO operation is done, it continues where it stopped.
unknown|5 years ago
[deleted]
nurettin|5 years ago
pirsquare|5 years ago
ram_rar|5 years ago
samirsd|5 years ago
heliodor|5 years ago
unknown|5 years ago
[deleted]