PyFilesystem – A Python interface to filesystems of all kinds

PyFilesystem is a Python module I started some time in 2008, and since then it has been very much a part of my personal standard library. I’ve used it in personal and professional projects, as have many other developers and organisations.

Recap

If you aren’t familiar with PyFilesystem; it’s an abstraction layer for filesystems. Essentially anything with files and directories (hard-drive, zip file, ftp server, network filesystems etc.) may be wrapped with a common interface. With it, you can write code that is agnostic as to where the files are physically located.

Here’s a quick example that recursively counts the lines of code in a directory:

def count_python_loc(fs):
    """Count non-blank lines of Python code."""
    count = 0
    for path in fs.walk.files(filter=['*.py']):
        with fs.open(path) as python_file:
            count += sum(1 for line in python_file if line.strip())
    return count

from fs import open_fs
projects_fs = open_fs('~/projects')
print(count_python_loc(projects_fs))

The fs argument to count_python_loc is an FS object, which encapsulates everything you would need to do with a filesystem. Because of this abstraction, the same code will work with any filesystem. For instance, counting the lines of code in a zip file is a single line change:

projects_fs = open_fs('zip://projects.zip')

https://www.willmcgugan.com/blog/tech/post/announcing-pyfilesystem-2/

Amazon States Language

This document describes a JSON-based language used to describe state machines declaratively. The state machines thus defined may be executed by software. In this document, the software is referred to as “the interpreter”.

Copyright © 2016 Amazon.com Inc. or Affiliates.

The operation of a state machine is specified by states, which are represented by JSON objects, fields in the top-level “States” object. In this example, there is one state named “Hello World”.

{
    "Comment": "A simple minimal example of the States language",
    "StartAt": "Hello World",
    "States": {
    "Hello World": { 
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloWorld",
      "End": true
    }
  }
}

When this state machine is launched, the interpreter begins execution by identifying the Start State. It executes that state, and then checks to see if the state is marked as an End State. If it is, the machine terminates and returns a result. If the state is not an End State, the interpreter looks for a “Next” field to determine what state to run next; it repeats this process until it reaches a Terminal State (Succeed, Fail, or an End State) or a runtime error occurs.

In this example, the machine contains a single state named “Hello World”. Because “Hello World” is a Task State, the interpreter tries to execute it. Examining the value of the “Resource” field shows that it points to a Lambda function, so the interpreter attempts to invoke that function. Assuming the Lambda function executes successfully, the machine will terminate successfully.

A State Machine is represented by a JSON object.

https://states-language.net/spec.html

Amazon Pinpoint – Hit your Targets with AWS

Today we are launching Amazon Pinpoint, a new service that makes it easy to run targeted campaigns to improve user engagement. Pinpoint helps you understand your users’ behavior, define who to target, what messages to send, when to deliver them, and tracks the results of the campaign.

Pinpoint enables real-time analytics with dashboards for analyzing user engagement, monetization, user demographics, custom events, and funnels so you can understand how users engage with your application. You can analyze and understand your user data by drilling down based on the segments you’ve defined, segmentation attributes, or time.

With Pinpoint, you can define target segments from a variety of different data sources. You can identify target segments from app user data collected in Pinpoint. You can build custom target segments from user data collected in other AWS services such as Amazon S3 and Amazon Redshift, and import target user segments from third party sources such as Salesforce via S3.

Once you define your segments, Pinpoint lets you send targeted notifications with personalized messages to each user in the campaign based on custom attributes such as game level, favorite team, and news preferences for example. Amazon Pinpoint can send push notifications immediately, at a time you define, or as a recurring campaign. By scheduling campaigns, you can optimize the push notifications to be delivered at a specific time across multiple time zones. For your marketing campaigns Pinpoint supports Rich Notifications to enable you to send images as part of your campaigns. We also support silent or data notifications which allow you to control app behavior and app config on the background.

Once your campaign is running, Amazon Pinpoint provides metrics to track the impact of your campaign, including the number of notifications received, number of times the app was opened as a result of the campaign, time of app open, push notification opt-out rate, and revenue generated from campaigns. You can also export the resulting event data and run custom analytics using your existing analytics systems. You can also A/B test different messages, track results, and then send the best message to your target segment.

With Pinpoint there is no minimum fee, no setup cost and no fixed monthly cost based on your total user pool. You only pay for the number of users you target or collect events from, the messages you send, and events you collect, so you can start small and scale as your application grows.

Now lets take a look at how Pinpoint makes it easy to setup a campaign.

https://aws.amazon.com/blogs/aws/amazon-pinpoint-hit-your-targets-with-aws/

15 Years of Concurrency

In a Tale of Three Safeties, we discussed three kinds of safety: type, memory, and concurrency. In this follow-on article, we will dive deeper into the last, and perhaps the most novel yet difficult, one. Concurrency-safety led me to the Midori project in the first place, having spent years on .NET and C++ concurrency models leading up to joining. We built some great things that I’m very proud of during this time. Perhaps more broadly interesting, however, are the reflections on this experience after a few years away from the project.

I’ve tried to write this article about 6 times since earlier this year, and I’m thrilled to finally share it. I hope that it’s useful to anyone interested in the field, and especially anybody who is actively innovating in this area. Although the code samples and lessons learned are deeply rooted in C#, .NET, and the Midori project, I have tried to generalize the ideas so they are easily consumable regardless of programming language.

http://joeduffyblog.com/2016/11/30/15-years-of-concurrency/

tiny-dnn is a C++11 implementation of deep learning

  • reasonably fast, without GPU
    • with TBB threading and SSE/AVX vectorization
    • 98.8% accuracy on MNIST in 13 minutes training (@Core i7-3520M)
  • portable & header-only
    • Run anywhere as long as you have a compiler which supports C++11
    • Just include tiny_dnn.h and write your model in C++. There is nothing to install.
  • easy to integrate with real applications
    • no output to stdout/stderr
    • a constant throughput (simple parallelization model, no garbage collection)
    • work without throwing an exception
    • can import caffe’s model
  • simply implemented
    • be a good library for learning neural networks

https://github.com/tiny-dnn/tiny-dnn#examples