HN Reader

Wait a sec… if the TLS handshakes look different, would it be possible to have an nginx level filter for traffic that claims to be a web browser (eg chrome user agent), yet really is a python/php script? Because this would account for the vast majority of malicious bot traffic, and I would love to just block it.

Showhn at the time https://news.ycombinator.com/item?id=30378562

There are API’s that chrome provides that allows servers to validate whether the request came from an official chrome browser. That would detect that this curl isn’t really chrome.

It’d be nice if something could support curl’s arguments but drive an actual headless chrome browser.

Cool tool but it shouldn't matter whether the client is a browser or not. I feel sad that we need such a tool in the real world

Here are my thoughts in order:

1. Can impersonate Chrome and Firefox? But that's already possible by using the User Agent

2. Oh, maybe there's JavaScript code to detect the browser, and this cURL build can run it?

3. I'll check the repo to see if it names changes or I have to look for them myself.

4. Alright, so it does TLS handshakes the same way browsers do. That's it?

5. Well, I guess someone might need this, but that someone is not me.

I'm rooting for Ladybird to gain traction in the future. Currently, it is using cURL proper for networking. That is probably going to have some challenges (I think cURL is still limited in some ways, e.g. I don't think it can do WebSockets over h2 yet) but on the other hand, having a rising browser engine might eventually remove this avenue for fingerprinting since legitimate traffic will have the same fingerprint as stock cURL.

Did they also set IP_TTL to set the TTL value to match the platform being impersonated?

If not, then fingerprinting could still be done to some extent at the IP layer. If the TTL value in the IP layer is below 64, it is obvious this is either not running on modern Windows or is running on a modern Windows machine that has had its default TTL changed, since by default the TTL of packets on modern Windows starts at 128 while most other platforms start it at 64. Since the other platforms do not have issues communicating over the internet, so IP packets from modern Windows will always be seen by the remote end with TTLs at or above 64 (likely just above).

That said, it would be difficult to fingerprint at the IP layer, although it is not impossible.

This tool is pretty sweet in little bash scripts combo'd up with gnu parallel on red team engagements for mapping https endpoints within whatever scoped address ranges that will only respond to either proper browsers due to whatever, or with the SNI stuff in order. Been finding it super sweet for that. Can do all the normal curl switches like -H for header spoofing

I had to do something like this with Ansible's get_url module once.

Was having issues getting module to download an installer from a vendors site.

Played with Curl/WGET, but was running into the same, while it worked from a browser.

I ended up getting both Curl + get_url to work by passing the same headers my browser sent such as User-Agent, encoding, etc

"For these reasons, some web services use the TLS and HTTP handshakes to fingerprint which client is accessing them, and then present different content for different clients."

Examples: [missing]

Only three patches and shell wrappers, this should get Daniel coding. Imho this should definitely be in mainline curl.

Kudos to the coder and the poster. I'm involved in a browser project that runs on OpenSSL, and figured I'd have to dig through WireShark myself at some point to figure this stuff out. Well, I may still need to, but now have many points of reference. If the most common use of OpenSSL is Python, then in the age of Cloudflare, a Firefox TLS spoofing option isn't just a good idea, it's a necessity.

I do kind of yern for the simpler days when if a website didn't mind bots it allowed it and if they did they blocked your user agent.

ive been using puppeteer to query and read responses from deepseek.com, it works really well but i have to use a stealth mode and "headed" version to make it think its a person

Good luck getting past imperva

If you thought cloudflare challenge can be bad, imperva doesn't even want most humans through

I'm always ambivalent about things like this showing up here. On one hand, it's good to let others know that there is still that bit of rebelliousness and independence alive amongst the population. On the other hand, much like other "freedom is insecurity" projects, attracting unwanted attention may make it worse for those who rely on them.

Writing a browser is hard, and the incumbents are continually making it harder.

Now I'm waiting for the MCP version of this.. :)

Set a UA and any headers and/or cookies with regular cURL compiled with HTTP/3. This can be done with wrapper scripts very easily. 99.999% of problems solved with no special magic buried in an unclean fork.

The notion of real-world TLS/HTTP fingerprinting was somewhat new to me, and it looks interesting in theory, but I wonder what the build's use case really is? I mean you have the heavy-handed JavaScript running everywhere now.

There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate

There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi