Nginx: a caching, thumbnailing, reverse proxying image server?

“A month or two ago, I decided to remove Varnish from my site and replace it with Nginx’s built-in caching system. I was already using Nginx to proxy to my Python sites, so getting rid of Varnish meant one less thing to fiddle with. I spent a few days reading up on how to configure Nginx’s cache and overhauling the various config files for my Python sites (yes, irony). In the course of my reading I bookmarked a number of interesting Nginx modules to return to, among them the Image Filter module.

I thought it would be neat to combine Nginx’s reverse proxying, caching, and image filtering to create a thumbnailing server for my images hosted on S3. If you look closely at the <img> tag below (and throughout this site), you can see Nginx in action…”

http://charlesleifer.com/blog/nginx-a-caching-thumbnailing-reverse-proxying-image-server-/

Stack Overflow: The Architecture – 2016 Edition

“To get an idea of what all of this stuff “does,” let me start off with an update on the average day at Stack Overflow. So you can compare to the previous numbers from November 2013, here’s a day of statistics from February 9th, 2016 with differences since November 12th, 2013:

  • 209,420,973 (+61,336,090) HTTP requests to our load balancer
  • 66,294,789 (+30,199,477) of those were page loads
  • 1,240,266,346,053 (+406,273,363,426) bytes (1.24 TB) of HTTP traffic sent
  • 569,449,470,023 (+282,874,825,991) bytes (569 GB) total received
  • 3,084,303,599,266 (+1,958,311,041,954) bytes (3.08 TB) total sent
  • 504,816,843 (+170,244,740) SQL Queries (from HTTP requests alone)
  • 5,831,683,114 (+5,418,818,063) Redis hits
  • 17,158,874 (not tracked in 2013) Elastic searches
  • 3,661,134 (+57,716) Tag Engine requests
  • 607,073,066 (+48,848,481) ms (168 hours) spent running SQL queries
  • 10,396,073 (-88,950,843) ms (2.8 hours) spent on Redis hits
  • 147,018,571 (+14,634,512) ms (40.8 hours) spent on Tag Engine requests
  • 1,609,944,301 (-1,118,232,744) ms (447 hours) spent processing in ASP.Net
  • 22.71 (-5.29) ms average (19.12 ms in ASP.Net) for 49,180,275 question page renders
  • 11.80 (-53.2) ms average (8.81 ms in ASP.Net) for 6,370,076 home page renders…”

http://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/

Building Large Flask Apps In The Real World

“Scaling a Flask application is no immediately obvious matter. At plug.dj we had ~22,000 line Flask application. At my previous employer our Flask application was significantly larger. Ultimately scaling a code-base is less about the framework used and more about the software design experience of the developers working on it. Scaling in terms concurrent users also has little to do with the web framework and more to do with your understanding of load-balancing, caching, databases, etc. That being said, what have I learned about how to organize a Flask application to comfortably grow?…”

https://etscrivner.github.io/posts/2014/10/building-large-flask-apps-in-the-real-world/

TensorFlow Serving

“Machine learning powers many Google product features, from speech recognition in the Google app to Smart Reply in Inbox to search in Google Photos. While decades of experience have enabled the software industry to establish best practices for building and supporting products, doing so for services based upon machine learning introduces new and interesting challenges.

Today, we announce the release of TensorFlow Serving, designed to address some of these challenges. TensorFlow Serving is a high performance, open source serving system for machine learning models, designed for production environments and optimized for TensorFlow…”

http://googleresearch.blogspot.com.br/2016/02/running-your-models-in-production-with.html
http://tensorflow.github.io/serving/

mpmath is a free (BSD licensed) Python library for real and complex floating-point arithmetic with arbitrary precision

mpmath is a free (BSD licensed) Python library for real and complex floating-point arithmetic with arbitrary precision. It has been developed by Fredrik Johansson since 2007, with help from many contributors.

The following example computes 50 digits of pi by numerically evaluating the Gaussian integral with mpmath. See 100 mpmath one-liners for pi and the documentation links below for many more examples!

>>> from mpmath import mp
>>> mp.dps = 50
>>> print(mp.quad(lambda x: mp.exp(-x**2), [-mp.inf, mp.inf]) ** 2)
3.1415926535897932384626433832795028841971693993751

mpmath runs on Python 2.5 or higher (including Python 3.x), with no other required dependencies. It can be used as a library, interactively via the Python interpreter, or from within the SymPy or Sage computer algebra systems which include mpmath as standard component. SageMathCloud lets you use mpmath directly in the browser…”

http://mpmath.org/

Making robust software: the “Mars Code” talk from USENIX HotDep ’12

“At USENIX Hot Topics in System Dependability miniconference, 7 Oct 2012, in Hollywood, California, Dr. Gerard Holzmann of NASA/JPL presented how NASA/JPL writes and tests robust software that survives rigors of interplanetary travel, planetary entry, descent, and landing, and operation in another world.

His methods are state of the art, and if you need to build robust software, take heed…”

http://www.verticalsysadmin.com/making_robust_software/

Records: SQL for Humans™

Records is a very simple, but powerful, library for making raw SQL queries to most relational databases.

Just write SQL. No bells, no whistles. This common task can be surprisingly difficult with the standard tools available. This library strives to make this workflow as simple as possible, while providing an elegant interface to work with your query results.

Database support includes Postgres, MySQL, SQLite, Oracle, and MS-SQL (drivers not included)…”

https://github.com/kennethreitz/records
https://pypi.python.org/pypi/records/