Requestium is a python library that merges the power of Requests, Selenium, and Parsel into a single integrated tool for automatizing web actions.
Requestium adds independent improvements to both Requests and Selenium, and every new feature is lazily evaluated, so its useful even if writing scripts that use only Requests or Selenium.
- Enables switching between a Requests’ Session and a Selenium webdriver while maintaining the current web session.
- Integrates Parsel’s parser into the library, making xpath, css, and regex much cleaner to write.
- Improves Selenium’s handling of dynamically loading elements.
- Makes cookie handling more flexible in Selenium.
- Makes clicking elements in Selenium more reliable.
- Supports Chrome and PhantomJS.
All code examples below are labeled for reference. They are purely intended to provide examples of concepts. Most of them can be written in a much better way.
Headless Chrome is a way to run the Chrome browser in a headless environment. Essentially, running Chrome without chrome! It brings all modern web platform features provided by Chromium and the Blink rendering engine to the command line.
Why is that useful?
A headless browser is a great tool for automated testing and server environments where you don’t need a visible UI shell. For example, you may want to run some tests against a real web page, create a PDF of it, or just inspect how the browser renders an URL.
Without your consent most major web platforms leak whether you are logged in. This allows any website to detect on which platforms you’re signed up. Since there are lots of platforms with specific demographics an attacker could reason about your personality, too.
For the sake of simplicity, let’s assume your everyday framework is React but if you use something else, replace React with X framework because this will still apply to you too.
“This tool turns a curl command into Go code. (To do the reverse, check out sethgrid/gencurl.) Currently, it knows the following options: -d/–data, -H/–header, -I/–head, -u/–user, and -X/–request. It also understands JSON content types. As always, be sure to go fmt your code and pay attention to imports. There’s probably bugs; please contribute on GitHub! For a quick way to generate a Go struct from JSON, see JSON-to-Go…”
“Google, Microsoft, Mozilla, and a few other people have been secretly toiling away in a new W3C WebAssembly Community Group, and what they have been working on is no small thing.
For more depth, see the follow-up article, “Why We Need WebAssembly: An Interview with Brendan Eich”…”
“WebSockets provides simple APIs to broadcast information to clients and simple APIs to ship information from the clients to the web server.
A realtime channel to send information from the server to the client is very welcome. In fact it is a part of HTTP 1.1.
However, a brand new API for shipping information to the server from web browsers introduce a new decision point for developers:
- When a user posts a message on chat, do I make a RESTful call and POST a message or do I bypass REST and use WebSockets?
- If I use the new backchannel, how do I debug it? How do I log what is going on? How do I profile it? How do I ensure it does not slow down other traffic to my site? Do I also expose this endpoint in a controller action? How do I rate limit this? How do I ensure my background WebSocket thread does not exhaust my db connection limit?…”
“What could be simpler than returning HTTP status codes? Did the page render? Great, return
200. Does the page not exist? That’s a
404. Do I want to redirect the user to another page?
302, or maybe
Life is bliss, well… until someone tells you you’re not doing this REST thing. Next thing you know, you can’t sleep at night because you need to know if your new resource returns the RFC-compliant, Roy-Fielding-approved status code. Is it just a
200 here? Or should it really be a
204 No Content? No, definitely a
202 Accepted… or is that a
What complicates matters is that the official HTTP/1.1 guidelines — the RFC — was originally written in 1997.† That’s the year you went surfing the cyberweb in Netscape Navigator on your 33.6kbps modem. It’s a little like trying to apply Sun Tzu’s Art of War to modern business strategy. Timeless advice, to be sure, but I haven’t yet figured out how The Five Ways to Attack With Fire are going to help me do market validation…”
Choosing an HTTP Status Code