Vincent's Blog

In short

• Fapws4 now follows the WSGI specification (PEP 3333) on both sides:
  what the application receives in environ, and what it sends back
  through start_response.
• A request with a non-UTF-8 byte can no longer leave the server
  answering 500 until it is restarted.
• Routes with accents or spaces work, whatever way the browser encodes
  them.
• The server stops cleanly on Ctrl-C or SIGTERM instead of aborting.
• There is now a test suite: sh make.sh test runs 96 functional checks.
• Some of this changes what an application sees. The list is at the end
  of this post.

Where it started: one bad byte

The trigger was a bug that is easy to describe. A request whose path
contained a raw non-UTF-8 byte, GET /\xff for instance, failed inside
the server and left a Python exception pending. Every following request
then failed too, so the server answered 500 to everybody until someone
restarted it. One request from anybody was enough.

The fix has two parts. Error paths now always clear a pending Python
exception. And such a request no longer fails at all, because the server
stopped guessing the encoding of what it receives.

Bytes in, bytes out: latin-1 everywhere

PEP 3333 has a precise answer to the encoding question: environ values
are strings holding the raw bytes of the request, decoded as latin-1.
Latin-1 cannot fail on any byte, and the application decides what the
bytes mean.

Since 0.7 this is how PATH_INFO, QUERY_STRING, fapws.uri and the
HTTP_* values arrive. Since 0.8 it is also true for SCRIPT_NAME, and
for the other direction: the status line and the response headers are
encoded as latin-1.

For plain ASCII nothing changes. For a path such as /caf%C3%A9, the
application receives '/café' and recovers the text itself:

path = environ['PATH_INFO'].encode('latin-1').decode('utf-8')

The benefit of doing both sides the same way is that a value copied from
the request into a header, a redirect to the current path for instance,
goes out byte for byte as it came in.

fapws.params is the exception, on purpose. It is the query string
already parsed by the C code into {key: [values]}, decoded as UTF-8 and
ready to use: ?page=caf%C3%A9 gives {'page': ['café']}.

start_response, as the specification describes it

start_response was the second area that did not match the
specification:

• Calling it twice used to send two sets of headers. A new call now
  replaces the previous one, and exc_info is honoured.
• It returns the write() callable the specification asks for.
• A generator may call it as late as just before its first yield. The
  server used to build the headers too early and sent a default
  200 OK without the application's headers.
• A call that fails halfway, on a malformed header for instance, now
  changes nothing. Before, the headers already processed leaked into
  the error page the application sent next.
• Printing the start_response object for debugging no longer blocks
  the calls that follow.
• The response's close() method is called exactly once on every path,
  including when the response cannot be started.

Routes with accents and spaces

Routes are now compared with the percent-decoded path, without the query
string. A route written '/café/' in PATHS is reached by
/caf%C3%A9/, which is what a browser sends, and '/my page' by
/my%20page. Before, such routes only matched clients sending raw
UTF-8, and the part removed from PATH_INFO could have the wrong length.

In a path, + is no longer turned into a space. That convention belongs
to query strings only, and it made a file named c++.txt impossible to
serve.

Static files

Staticfile, the helper serving files from a directory, got a round of
fixes:

• max-age was an absolute timestamp, about 57 years, instead of a
  duration.
• A file of unknown type was sent with Content-Type: None; it is now
  application/octet-stream.
• The ETag is quoted as HTTP requires, and its weak form is accepted.
• A path is refused only when a segment is exactly .., so a..b.txt
  is served again.
• File names with non-ASCII characters are found, and a name that is not
  valid UTF-8 gives a 404 instead of a 500.

A clean shutdown

Stopping the server with Ctrl-C or SIGTERM used to end in a libuv
assertion and an abort. Python was not finalised, so atexit handlers
did not run.

The server now stops accepting connections, finishes the responses it
has started, and exits with status 0. A request that is not completely
received yet is dropped. If a response takes too long, a further signal
ends the server at once.

Smaller fixes

• Data passed to write() and items of a list of 4 GiB or more are
  refused with a 500. They used to be sent truncated, without any error.
• A request body above 64 KiB is spooled to a file; its wsgi.input is
  now closed when the request is over.
• write() called from inside the response generator is refused, as the
  specification demands, with a message saying what to do instead.
• The wiki sample no longer decodes page names twice.
• make.sh is plain POSIX shell, honours CC and PREFIX, passes the
  BSD include and library paths, and stops on the first error.

A test suite

Until now Fapws4 had no automated tests. It has one:

sh make.sh test

This compiles the server, starts it on a free local port with a test
application, sends it real HTTP requests over sockets and compares the
answers. It ends by stopping the server and checking its exit status and
its log. Only Python's standard library is needed and nothing is
installed.

There are 96 checks in 11 groups: static files, headers, the different
kinds of responses, the close() contract, routes, start_response,
size limits, request bodies, malformed and aborted requests, a burst of
requests, and the shutdown. Every check is reported, passed or failed,
with a summary per group.

Before releasing 0.8 I also ran the suite under Python's debug allocator
and under the C library's heap checks, and pushed about 300,000 requests
through the server with ab: no failed request, and memory and file
descriptors stayed flat.

How the work was done

Most of this came out of code reviews done with Claude Code. Each finding
became one small commit, with its own test and a message explaining the
cause and the consequences; I read every diff before it went in. The
second review, run on the release itself, found four inaccuracies in the
release notes, which are corrected.

One finding turned out not to be a bug: refusing write() inside a
generator is what PEP 3333 asks for. Only the error message was improved.

Upgrading

sh make.sh install now compiles before installing, so the executable
and base.py always come from the same sources. They must be updated
together.

If you come from 0.6, check these points in your application:

• Environ values are latin-1 strings (0.7): PATH_INFO,
  QUERY_STRING, fapws.uri, HTTP_*, and SCRIPT_NAME (0.8).
  Re-decode them if you expect non-ASCII text. fapws.params is not
  affected.
• Response headers are encoded as latin-1 (0.8). Real non-ASCII text
  must be converted first with .encode('utf-8').decode('latin-1'); a
  character above U+00FF is answered with a 500.
• Routes are matched on the decoded path (0.8). Write them as plain
  text in PATHS, not percent-encoded. An encoded form of an ASCII
  route, /%61dd, now reaches '/add'.
• + in a path is a plus (0.8). Use %20 for a space.
• start_response called twice replaces the headers (0.7), and
  calling it after the headers are sent raises.
• Generators (0.8): an exception before the first yield gives a
  500 instead of an empty 200, and the first item is produced before
  the headers are sent.
• Staticfile (0.7): the ETag format changed, so browsers fetch
  each cached file once more.
• wsgi.input (0.8) cannot be read after the request has ended when
  the body was large.

The README has the same list in its "Upgrading from 0.7" section.

Getting it

got clone ssh://anon@repo.vincentdelft.be/fapws4
got checkout fapws4.git
cd fapws4
sh make.sh test
sh make.sh install

You need Python 3.8 or newer, libuv and a C compiler. Feedback and bug
reports are welcome.

It compiles and runs on OpenBSD 7.9 and voidlinux

This blog is served by Fapws4 since 2019