If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases

That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using multiprocessing.Pool and getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.

And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.

What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.

Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.

At least we agree on something!

If you didn’t use a profiler, then how can you be sure that the issue was Python and not something else? If the slowness really is evenly distributed through the code as you suggest, then a profiler would show that almost instantly and it’s very easy to do that in Python using cProfile.

On top of that, the Python standard library is built for convenience, not performance. The nature of it being a standard library means that the implementation has to be simple. fastregex uses LLVM and SIMD for example, that’s something that’s wildly impractical for the Python stdlib to support. In contrast, I have no idea what regex or JSON library you used for your Rust code, but if it were outside the stdlib, chances are high that it had much more leeway to optimize performance because it didn’t have all the other maintenance obligations that stdlib code needs to support.

All of this is just pure speculation on my part, but ultimately I’m trying to make a point that it is hard to say that the difference was caused by a language issue and not something else. There are lots of other things that could be contributing to this difference and you can’t really say for sure without meaningful diagnostics.

As an example of this, look at Nicholas Nethercote’s excellent series of blogs on making the Rust compiler faster: https://nnethercote.github.io/

The “bottlenecks” they find result in 1% speedups.

The kinds of software that you and I write on a daily basis is nothing compared to pieces of software that get millions of man-hours per year. Assuming you’re not Linus Torvalds or Ted Ts’o hiding behind a funny alias, of course. I don’t think anyone is suggesting that it’s a good idea to write a compiler in Python (outside of prototyping purposes perhaps) since it’s obviously not I/O bound, but if you and I are trying to optimize something we wrote ourselves, we’re more likely to be in the “nice small hotspot” domain you described, rather than the 1% bottleneck speedups discussed in the link you posted.

Very cool link though by the way, I look forward to reading as much of it as I can…

Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.

I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.

I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.

I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.

Sure, it’s slow. Google’s landing page is 4 MB for a small graphic, a text form, and some buttons. But what are you going to do? Not use it? They won. They don’t have to fix shit.

This neglects to include the massive amount of additional front-end code Google (and other large tech companies) toss on their webpages for things that are purely for their own benefit while being of zero or negative value to the users. Mouse tracking, analytics, fingerprinting, and all kinds of other harmful content that tech companies put out go a long way towards reaching that 4MB figure.

It’s not so much “But that’s not how you build stable software.” It’s more like “But that’s not how you build software [at all today].” Modern software is powered by loads of shared packages, that’s just how all modern languages work. This is equally true for Rust and Python.

You’re not wrong - all libraries with automated dependency packaging do have this problem. But you’re downplaying the scale by a lot. Javascript dependency graphs are absolutely massive. Python has a decent standard library that reduces the number of shared packages by quite a bit. There are definitely downsides to having a large standard library like Python has, but it does significantly reduce the probability of a supply chain attack compared to Javascript.

I can’t comment on Rust personally, since I don’t have much experience with the ecosystem.

Or the fact that you can’t do concurrency especially across the language barrier from another language without multiple python runtimes orchestrated with a queue and IPC because of possible process-wide global state and similar concerns/assumptions for any python library dependency that is not all native across its own dependency chain?

This is very out of date. I haven’t written Python for over 4 years and even back then, I was using multiprocessing.Pool for pretty much all parallel processing that I ever needed. Sure, it’s doesn’t fit every use case for concurrency, but for simple stuff it’s perfectly fine.

Not only that, it’s one of the simplest parallel processing APIs I’ve ever used:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

Did you use a profiler to find the bottlenecks? Did you try using cjson to speed up the json parsing? I would also avoid using the stdlib regex library too. It’s probably more of a convenience/reference implementation, not something to be intended for high performance. Maybe there’s a re2 wrapper for Python or a similar regex library in C with Python bindings.

Rewriting in Rust is fine, but good Python devs know how to find bottlenecks in their software and replace them with fast implementations. I’ve written plenty of performant Python code, it’s not very hard to profile and optimize. Usually the performant stuff is already written in C/C++ or Fortran anyway (numpy, PyQt, etc.)

Just be grateful that you don’t have to pay for the privilege of watching ads… yet.

Cable TV, I’m looking at you!

The end may be peaceful, but he didn’t say how rough the journey could be!

wtf are you torrenting now, coffee pot???

I have a friend in Ontario, and it’s quite a nice place to visit and seems so beautiful!

Then I read about Doug Ford and… yeah.

Canadians are such lovely people though.

Not surprising at all. Spez made it clear that he wanted to kill the site during the API shutdown, we all knew that any exceptions to the rule would be short-lived, which is why I cut them off cleanly and never bothered going back. Alternative front-ends may well continue to exist the same way Twitter front-ends still do, but why swim upstream? Just let it die.

Fediverse is just so much nicer anyway. Reddit may have more volume, but most of it is shit. The discussion here may not be as big, but it’s just such better quality.

I’m tired of endlessly stressing out about money, so I would say somewhere with a lower cost of living (but still relatively safe and healthy) and with nice friendly people. If I had to guess where that’d be, I’d say Thailand/Vietnam, Costa Rica, or South America. Maybe even Spain too, but outside of the major cities? I’ve heard that the north coast or Extramadura can be really nice too…

It’s fun to dream about, but not something I think about seriously. All places have their pros and cons, all places have good and bad people. I would definitely love to at least travel there someday though!

Eventually, all these tech companies scale up to a size where they don’t have to give a shit about people anymore. They only care if you’re a celebrity or somehow manage to create a big PR show that they have to respond to. But at this point, regular people are just like blades of grass that will eventually get mowed down somehow and there’s nothing you can do about it.

Now it’s dead and time to move on.

So true. Employers hate labour so much, it can only be because they’re shit people.

Ramps benefit all of us, they’re not just for wheelchairs. Have you ever had to push around a moving cart or a stroller? You don’t need a disability, you’ll still really appreciate the ramps!

Perhaps the reason he has that kind of money is because the people that give it to him know that he doesn’t have principles to begin with.

And they say AI doesn’t cause brainrot… “Sorry you can’t get AI to work for you. Keep trying!” sounds like the kind of thing you would expect from someone who only outputs the highest quality of work, doesn’t it?

I’m not saying all AI users are like this, and there are many smart developers that do use it effectively (like Torvalds), but there is definitely a loud group of AI cheerleaders that extremely careless and overconfident, while lacking any sense of self-criticism or oversight of what they say or do.

And by the way, you definitely can build simple web-apps in a day without AI. I was doing that in 2015 by just following the Django tutorial. It took me longer than a day because I tried to read the documentation and understand it. But if I put the same amount of effort that these low-effort AI users do, I could have easily just copy-pasted shit from the finished tutorial and been done in an hour with the same level of understanding (or lack thereof) that AI users have.

That’s the difference between good and bad engineers - you have the tools in front of you, but can you make yourself a better engineer while you use them?

Speaking of which, you know what these AI tools do? They just go to github and copy off some project that already does what you want it to do. Anyone could do that before AI existed. Sure, it’s definitely impressive that they can understand what you want to do and find the right result (usually), but you could do that ever since search engines were invented. AI is just the “I’m feeling lucky” button from Google from back when people had enough sense to be a little critical of the results they were looking at before they clicked it.

Article is light on details. I’m curious how they define what is “algorithm”-generated content and what isn’t.

Ad personalization is hard to verify because it’s all happening on the backend. What you see in a feed is much more visible, so maybe that will make it harder to work around. I haven’t read much about how the legislation works and how it defines algorithms, but if it’s something like, “only show me people I’ve subscribed to”, then it would be pretty easy to verify whether they are showing that or not.

But in general, I agree. Big tech companies always find a way to work around this legislation. Enforcement is the issue, not the laws themselves.