Funny you mention that: https://www.panamericandispatch.org/p/meta-ignores-brazilian-court-order

Really hope Facebook gets pounded for their interference in Brazil’s election.

Great news, not just for Brazil, but for the whole world.

But winning the election is just the first step. The more important part is that he uses his victory to deliver the results that he promised. If politicians did this better, we would never see a far right movement in the first place.

Let’s hope we see it happen.

If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases

That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using multiprocessing.Pool and getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.

And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.

What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.

Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.

At least we agree on something!

If you didn’t use a profiler, then how can you be sure that the issue was Python and not something else? If the slowness really is evenly distributed through the code as you suggest, then a profiler would show that almost instantly and it’s very easy to do that in Python using cProfile.

On top of that, the Python standard library is built for convenience, not performance. The nature of it being a standard library means that the implementation has to be simple. fastregex uses LLVM and SIMD for example, that’s something that’s wildly impractical for the Python stdlib to support. In contrast, I have no idea what regex or JSON library you used for your Rust code, but if it were outside the stdlib, chances are high that it had much more leeway to optimize performance because it didn’t have all the other maintenance obligations that stdlib code needs to support.

All of this is just pure speculation on my part, but ultimately I’m trying to make a point that it is hard to say that the difference was caused by a language issue and not something else. There are lots of other things that could be contributing to this difference and you can’t really say for sure without meaningful diagnostics.

As an example of this, look at Nicholas Nethercote’s excellent series of blogs on making the Rust compiler faster: https://nnethercote.github.io/

The “bottlenecks” they find result in 1% speedups.

The kinds of software that you and I write on a daily basis is nothing compared to pieces of software that get millions of man-hours per year. Assuming you’re not Linus Torvalds or Ted Ts’o hiding behind a funny alias, of course. I don’t think anyone is suggesting that it’s a good idea to write a compiler in Python (outside of prototyping purposes perhaps) since it’s obviously not I/O bound, but if you and I are trying to optimize something we wrote ourselves, we’re more likely to be in the “nice small hotspot” domain you described, rather than the 1% bottleneck speedups discussed in the link you posted.

Very cool link though by the way, I look forward to reading as much of it as I can…

Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.

I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.

I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.

I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.

Sure, it’s slow. Google’s landing page is 4 MB for a small graphic, a text form, and some buttons. But what are you going to do? Not use it? They won. They don’t have to fix shit.

This neglects to include the massive amount of additional front-end code Google (and other large tech companies) toss on their webpages for things that are purely for their own benefit while being of zero or negative value to the users. Mouse tracking, analytics, fingerprinting, and all kinds of other harmful content that tech companies put out go a long way towards reaching that 4MB figure.

It’s not so much “But that’s not how you build stable software.” It’s more like “But that’s not how you build software [at all today].” Modern software is powered by loads of shared packages, that’s just how all modern languages work. This is equally true for Rust and Python.

You’re not wrong - all libraries with automated dependency packaging do have this problem. But you’re downplaying the scale by a lot. Javascript dependency graphs are absolutely massive. Python has a decent standard library that reduces the number of shared packages by quite a bit. There are definitely downsides to having a large standard library like Python has, but it does significantly reduce the probability of a supply chain attack compared to Javascript.

I can’t comment on Rust personally, since I don’t have much experience with the ecosystem.

Or the fact that you can’t do concurrency especially across the language barrier from another language without multiple python runtimes orchestrated with a queue and IPC because of possible process-wide global state and similar concerns/assumptions for any python library dependency that is not all native across its own dependency chain?

This is very out of date. I haven’t written Python for over 4 years and even back then, I was using multiprocessing.Pool for pretty much all parallel processing that I ever needed. Sure, it’s doesn’t fit every use case for concurrency, but for simple stuff it’s perfectly fine.

Not only that, it’s one of the simplest parallel processing APIs I’ve ever used:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

Did you use a profiler to find the bottlenecks? Did you try using cjson to speed up the json parsing? I would also avoid using the stdlib regex library too. It’s probably more of a convenience/reference implementation, not something to be intended for high performance. Maybe there’s a re2 wrapper for Python or a similar regex library in C with Python bindings.

Rewriting in Rust is fine, but good Python devs know how to find bottlenecks in their software and replace them with fast implementations. I’ve written plenty of performant Python code, it’s not very hard to profile and optimize. Usually the performant stuff is already written in C/C++ or Fortran anyway (numpy, PyQt, etc.)

You wrote, “A complete lack of specific action suggested.” so I provided some specific action. I don’t like being an asshole, but people tend to act like assholes if they think the people they’re talking to are also assholes. I’ll be more polite if you’re willing to do the same.

In any case, what I wrote is, “Do whatever you want, vote however you want, but real change won’t come until you throw out that stinking piece of trash constitution” and somehow, you assumed that it means, “don’t even try not even in the primaries.” I have no idea how you made that mental connection. You asked how to fix it, and I told you how to get it done. You wanted a direct example, and I posted several of them. I even found a guy that’s already started the process, just go and read the wiki link I posted on Lessig. What more do you want?

There are dozens of ways it could be fixed. Here, I’ll do the thinking for you.

First of all, if you read your own constitution, you would see that there are multiple ways to amend the constitution:

The Congress, whenever two thirds of both Houses shall deem it necessary, shall propose Amendments to this Constitution, or, on the Application of the Legislatures of two thirds of the several States, shall call a Convention for proposing Amendments, which, in either Case, shall be valid to all Intents and Purposes, as Part of this Constitution, when ratified by the Legislatures of three fourths of the several States, or by Conventions in three fourths thereof, as the one or the other Mode of Ratification may be proposed by the Congress; Provided that no Amendment which may be made prior to the Year One thousand eight hundred and eight shall in any Manner affect the first and fourth Clauses in the Ninth Section of the first Article; and that no State, without its Consent, shall be deprived of its equal Suffrage in the Senate.[3]

source

There are two ideas right off the bat: either start writing amendments and getting them passed through Congress, or have the state legislatures demand a convention be arranged. You don’t even need your federal Congress to get involved if you use the second approach. Did you already know that? (And since you think voting is so effective, you can invoke this holy power against the politicians that don’t support it.)

See, here’s a guy that already tried to start the process, maybe you can jump on board!

In 2010, Lessig began to organize for a national Article V convention.[81] He co-founded Fix Congress First! with Joe Trippi.[82] In a speech in 2011, Lessig revealed that he was disappointed with Barack Obama’s performance in office, criticizing it as a “betrayal”, and he criticized the president for using “the (Hillary) Clinton playbook”.[83] Lessig has called for state governments to call for a national Article V convention,[84] including by supporting Wolf-PAC, a national organization attempting to call an Article V convention to address the problem.[85]

https://en.wikipedia.org/wiki/Lawrence_Lessig?useskin=vector#Article_V_convention

Or maybe you need to find a not-so-legal approach. Your constitution was written because the previous one (Articles of the Confederation) were not so effective. How is your current situation today different from the one then? They figured out a way to get it done, and they were under far less duress then than you are now so… why can’t you? You start writing the new one, then ratify it the same way the same way the old one was.

And in case you weren’t aware, this happens all the time! Look at France. France is on their Fifth republic, the most recent being established in 1958, according to Wikipedia. And they’re not the only ones. There are new constitutions being proposed and voted on all over the world all the time, in the Middle East, in Africa, etc. Read the news and you’ll find out more about it. I recommend Deutsche Welle, they have excellent coverage of global affairs.

There you go. Hope you enjoyed a small lesson about civics in your own country from a foreigner. There are plenty of ideas to get you started, and now that you’re more in the know, don’t complain if voting by the rules doesn’t bring about the change you’re looking for. Good luck!

And another edit to add: yes, it’s hard work, but you have to start talking about real solutions at the very least. You can see very clearly and easily from where we are today that working within the system as it’s currently designed isn’t working and it probably never will. It’s up to you to pick whatever path you think is best, but I think it’s obvious that voting in your shitty electoral process just isn’t going to work.

I’ll probably get downvoted to hell for saying this, but whatever, it needs to be said. It’s just unbearable listening to how all of you defend voting for people you hate and doing nothing about it.

Here’s the truth: if you have one party that’s neo-fascist, and another party that does nothing, then in reality, your country is a one-party neo-fascist state.

What you need to do about it is fixed the root cause. Throw your constitution in the trash and write a new one that guarantees proportional representation. Sure, PR isn’t perfect, but it’s better than a single party state.

No, voting in primaries isn’t going to fix the problem. Maybe you get a small handful of 10 politicians around the country that make you feel good about yourselves, but there are still thousands more standing in their way preventing any real progress. What have Bernie Sanders or AOC really achieved? Nothing, because there’s 500 other people in Congress. What has Mamdani done for more than the tiny fraction of Americans living in NYC? Nothing, because a mayor in NYC is can’t do shit for the 90% of people who live in other parts of the country.

Do whatever you want, vote however you want, but real change won’t come until you throw out that stinking piece of trash constitution you have and write one that makes the USA a real democracy. Shout it out as loud as you can, because until then, nothing meaningful is ever going to change.

P.S. It’s not that people aren’t aware of the shitty system, it’s the fact that absolutely nobody is talking about the real problem, and there’s no movement whatsoever to fix it. Even Canadians are pushing desperately for PR and getting nowhere, but the USA is even more silent about it, and it’s fucking infuriating.