Automatic generation of rulesets

[galeksandrp]

https-everywhere-bot got mad?????

@J0WI @jeremyn

Submit app comparison (+sources)	Generators comparison (+sources)	Domain enumerators

Submit app comparison

Note that ✅ is clickable

Backend	Frontend	Prod	License	Dependencies	GitHub caps
galeksandrp@webhook	galeksandrp/httpse	✅	MIT	Travis GitHub Docker (galeksandrp@docker-httpse)
galeksandrp@webhook-2	galeksandrp@gh-pages	✅	MIT	Travis GitHub Docker (galeksandrp@docker-httpse)	public_repo
galeksandrp@webhook-client	galeksandrp/galeksandrp.github.io@master	✅	MIT	Travis GitHub Docker (galeksandrp@docker-httpse)	public_repo email

Generators support

Author/Link	@Foorack #6912 (comment)	gloomy-ghost/ew@master	galeksandrp@check-sublist3r	@pgerber #7706	Foorack/httpse-ruleset-generator@master	guiweber/ssl-yeah@master
galeksandrp/httpse	✔️		✅	✔️	✔️
galeksandrp@gh-pages	✔️		✅	✔️
galeksandrp/galeksandrp.github.io@master	✅		✅	✅

✔️ not deployed to prod
✅ deployed to prod

https://github.com/galeksandrp/httpse/issues/new

• No progress shown, but you can just wait when your PR will be arrived or see Travis log for specified *-auto branch in repo below.
• Incoming PRs will have mine as author. (fix can produce issues since github limit unanteficated requests to 60 per hour)
• Use github-email for discovering email.
• Single ruleset can't be PRed once. (branch will be rebased if existing branch have old test scripts)
• If ruleset is already present in HTTPSE and have different filename than lowercase.domain.xml PR will be created, but PR's Travis will fail.
• Issue number not passed to PR, so issue can't be automatically closed.
• Even empty ruleset will be PRed.
• ~~~Scan limit is 10m.~~~ Now 50m.
• Use shallow clone for speedup.
• Mention issue author in PR.
• Automatically comment && close issue when HSTS or no rules found.

https://galeksandrp.github.io/https-everywhere/

• Use OAuth to PR from issue author.
• Store token on server either local to bypass GitHub login attempts limit
• Do HSTS preload check from client side.
• Use Mozilla Public Suffix List for domain extracting
• Limited to 10 PR per hour.

Open PRs

• Check empty comment
• Rescan rulesets
• Rescan rulesets for mixed content

My mixed content position

Achived copy of this post in my repository: okyes/httpse#122 (comment)

[jeremyn]

Thanks @galeksandrp . I think you mean @J0WI (with a zero). Also @gloomy-ghost is a new maintainer.

[gloomy-ghost]

It's a good idea, and I personally have a python script to do the same job. I'm not good at bash programming, but here is a suggestion:

• Check http when https returns a non-2xx code.
• If https also returns a non-2xx code, try to automatically find a test on Google/Bing/DuckDuckGo/whatever. (I'm currently doing it by hand, which is boring)

[jeremyn]

I think @Foorack also has a script as well.

[numismatika]

In this context we also should revisit #2718 and #6912 as they are closely related.

[Foorack]

Good day. Yes, I have for some time working on a ruleset generator, Current version is completely written in python. It currently integrates with Sublis3r to automatically find domains. It can also check for MCB, SSL issues, cert issues and a bunch of other things. I have recently been taking all feedback I got to improve the program and I'm currently in the process of rewriting it again due to the recent complexity.

[Foorack]

Personally, I suggest we all should work on the same project. Both for code efficiency (stronger together) but also to save time. The goal would be to get the program into a state where it can cover nearly any domain. We currently have a golden situation with the vast amount of training data available for us. Ultimately I think it should be merged into the HTTPSE project. What is everyone's opinion on this?

Update: My two responses above was written on my phone. I only responded to jeremyn's comment and did not read the original issue description. As I understand it you want people to create an issue with the domain in the title. Your program will then generate a ruleset and push it to the HTTPSE repository. To be honest, I am not in favour of your approach for several reasons.

First, it will make up dependent on you and your system. Let's say if anything were to happen or you lost all interest; then we would be in a difficult situation. Secondly, your approach seems like it's closed-source, as I can't find anything about it on your profile. Best would be for it to be open source under a license like GPL where anyone can contribute to improving it and use it to generate rulesets on their own.

[galeksandrp]

@Foorack

Submit app comparison (+sources)	Generators comparison (+sources)	Domain enumerators

Actually, I don't really like this idea. In my opinion, a better thing would be a submission form allowing non-technical users to submit websites for rule creation. Automated rule generation is guaranteed to cause problems in complex cases, which are common.

[jeremyn]

As a maintainer, I would really prefer all of you who are using scripts work on them together and in the open. If you are going to take a role in a script writing process, please stay involved in discussions such as #6868 . If you are going to write a script that produces output that I need to review, then it should take into account how I review it, and I don't want to have separate, different conversations with each of you about that.

If you want a script reviewed, put the script up for code review, not a hundred pull requests from the script that all have the same problems. I would rather spend my time encouraging new contributors and resolving tricky issues than robotically reviewing a ton of script output from from experienced contributors. You should submit only a few pull requests at a time and your script should take feedback into account before you make new pull requests with it.

@galeksandrp In particular, you have put a lot of time into HTTPS Everywhere and obviously care about the project, and that's great. However you produce a lot of messy pull requests and then are slow about taking feedback. For the fnal.gov pull request that we discussed several times, I put a lot of time into an initial review, you didn't implement my feedback, and then I spent more time implementing my own changes onto your branch. I resubmitted your #5372 as my #7493 to get it off my list. Your example PR in this issue, #7692 , has a blank top comment, which is obviously wrong.

Anyone who submits a pull request should be willing to take feedback on that pull request in a timely way. I am very against an automatic rule submission process where a person can submit a messy pull request under someone else's name that a maintainer then has to clean up after.

[jeremyn]

@smw-koops For your #7691 (comment) , see what I wrote in #6322 (comment) .

[gloomy-ghost]

@jeremyn I think we can make reviewing of simple rulesets (only http -> https) automatically. We were working like bots in most of PRs, and I really don't like that.

Testing on travis-ci are conservative, and we can have some progress as reviewers that are more radical. The idea is to pass targets which are obviously okay, and filter out the others for further checking. For example, automatically finds MCB problems and throws decoding error, and reviewers can check these targets separately. Also we can set max_distance = 0.01, so we can check small difference between http and https.

[jeremyn]

@gloomy-ghost It should be possible but in reality, I guess well under 5% of the pull requests I review pass without having some problem. A valid script should be able to pass a ruleset Turing test, where the reviewer submits several flawless rulesets and only then reveals they were using a script to make them. That can't have happened yet because so few rulesets pass without a problem.

To clarify my earlier comment, I don't mind reviewing a ruleset just because it came from a script. I do mind reviewing hundreds of rulesets that all have similar problems, or a ruleset with an obvious problem (#7692 with the blank top comment), or a super complicated ruleset with lots of problems (#7504 (comment) for SLU.se).

I would like it to work like this:

1. A contributor writes a script and makes it available for community review.
2. They submit a ruleset with that script from some limited set of domains.
3. I find a problem with the ruleset.
4. The contributor either (a) manually fixes the ruleset, and I merge it, or (b) disagrees with my review and either I change my mind and merge it, or the contributor changes their mind and goes back to (a).
5. If (a), the contributor also updates the public script so it doesn't make the same mistake again.

Once enough rulesets are submitted without any problems found, then we can be confident enough to allow the script to automatically generate rulesets for whatever our limited set of domains is. We can then try expanding the ruleset to cover a larger set of domains.

In the above sequence, by "I find a problem", I mean any of the reviewers. If the contributor gets conflicting feedback from different reviewers, then we need to fix the conflict before moving forward. Similarly, by "a contributor", I mean any contributor working on the common script project.

[galeksandrp]

Submit app comparison (+sources)	Generators comparison (+sources)	Domain enumerators

Generators comparison

[galeksandrp]

Note that ✅ is clickable

Author / Link	License	Language	Dependencies	Readme/Launch script	Install info	Example	CI
@Foorack #6912 (comment)	custom nolicense	python3	python2.7 (aboul3la/Sublist3r@master)	✅	see below	✅	✅
gloomy-ghost/ew@master	GPLv3	python3	(gloomy-ghost/Sublist3r@1b12560)	see below	see below
galeksandrp@check-sublist3r	MIT	bash	python2 (aboul3la/Sublist3r@master https-everywhere-checker)	✅	#6912 (comment)	✅	✅
@pgerber #7706	GPLv2+	python3	python2 (aboul3la/Sublist3r@master)	#7706 (comment) #7706 (comment)	#7706 (comment)	#7706 (comment)	✅
Foorack/httpse-ruleset-generator@master	GPLv3	python3.4	(aboul3la/Sublist3r@ebe9a05)	✅	Foorack/httpse-ruleset-generator#1 (comment)	✅	✅
guiweber/ssl-yeah@master	GPLv3	node		✅	✅

Author / Link	Refused	Timeout	Bad cert	Wrong chain	SSL error	Redirect	Different content	Non 200	Mixed content
@Foorack #6912 (comment)	✅	✅	✅	✅	✅	✅	✅
gloomy-ghost/ew@master	✅	✅	✅		✅	✅		✅	✅
galeksandrp@check-sublist3r	✅	✅	✅		✅	✅	✅	✅	✅
@pgerber #7706	✅	✅	✅			✅		✅
Foorack/httpse-ruleset-generator@master	✅	✅	✅	✅	✅	✅	✅	✅	✅

Achived copy of this post in my repository: okyes/httpse#120 (comment)

[galeksandrp]

https://github.com/galeksandrp/https-everywhere/commit/check-sublist3r

• When no issues detected, produces an empty comment
• Produces comments for nonexisting domains.
• Does not check mixed content. Mixed check #8960

Bonus:

Author / Link	list of domains to STDIN	xml to STDOUT	list of domains from file	xml to file	Insert to existing xml	DNS zone as input	Check existing XML	Top comment as input
@Foorack #6912 (comment)				✅
gloomy-ghost/ew@master		✅
galeksandrp@check-sublist3r				✅			✅
@pgerber #7706	✅	✅	✅	✅	✅	✅	✅	✅
Foorack/httpse-ruleset-generator@master				✅

@Foorack #6912 (comment)

git clone https://gist.github.com/51893f7d0f8b16d6d8be25073131d4b6.git
chmod +x ct.sh
git clone https://github.com/aboul3la/Sublist3r.git
docker run --rm -ti -v $(pwd):/opt -w /opt electronicfrontierfoundation/https-everywhere-docker-base
apt-get update
apt-get install -y $(cat Sublist3r/requirements.txt | sed 's/^/python-/') python3-requests
./ct.sh europa.eu

gloomy-ghost/ew@master

git clone https://github.com/gloomy-ghost/ew.git
git submodule init
git submodule update
docker run --rm -ti -v $(pwd):/opt -w /opt electronicfrontierfoundation/https-everywhere-docker-base
ln -s Sublist3r/subbrute .
python3 sub.py

Achived copy of this post in my repository: okyes/httpse#121 (comment)