Notes on a New World Order

Would you like fries with your Universe?

My response to the FCC to proceeding 17-108
2017-05-30 09:50

Comments

Introduction

I submitted this as an official filiing to the FCC (confirmation number 20170530680421266) as my personal take on why the proposed changes are wrong for consumers and the industry. The full response is included below, you may want to have a copy of the proceeding open as well, as I go through and response to each paragraph (that I chose to respond to) by number.

My Filing

As a professional in the telecommunications industry, I find these proposed changes obnoxious to the public interest. The central assumption seems to be that a functioning market in Internet Service Providers exists, and has served us well for the past 20 years under the previous regulatory framework (before the Title II Order).

This is not the case, and policies that depend on market competition require a functioning market to be effective; that is, a market with a low cost of entry, a plethora of competing offerings (not just two or three) and low friction to changing which offering to buy. This is decidedly not the case with Internet Service in any part of the United States I am aware of. Wo while the past years have seen a meteoric rise in Internet services and innovation, it i smy opinion that this is in spite of the regulatory environment, rather than caused by it.

In Paragraph 27, the commission makes a grave and dangerous error in stating that a suer "posting on social media" or "drafting a blog" is engaged in making use of an information service provided by their broadband Internet provider. The information service is that of the social media service, or the blogging platform, not of the user's ISP. The only service of their ISP being used is the transmission of the data (which the user has chosen) to the point they have determined (the provider of the information service) without any modification.

In particular, the broadband provider does not "offer" the social media service to its subscribers, the social media company does so. As, likewise, the blogging platform offers their suers the "capability" to publish and make available information online. The broadband provider does not give or grant this capability by merely connecting their users' computer to the information services' computers — the provider of the blogging service is essential to the process.

By analogy with a traditional telephone system, calling a customer service number does not imply your telephone provider offers an information service because the call center provides an automated process for you to access your account information. The service provided by your phone company is telecommunications; the service provided by the call center is an information service.

This assessment is necessarily from the perspective of the user, as the text of the act specifically describes the classifications in terms of the offering — the service that is purchased.

"Can broadband users indeed access these capabilities?"

The emphasis on access is important; yes, users of broadband Internet services can access those capabilities — but those capabilities are not offered by their broadband provider.

"Are there other capabilities that a broadband Internet user may receive with service?"

Yes, substantially many other capabilities are permitted by broadband service today. These include direct interconnection over VPN or other remote access technologies, of the users' equipment across the Internet, a fully transparent telecommunications service. As well as access to service providers that provide transit and connectivity to other networks, like mobile SMS (usually via email gateways) or Voice over IP services that interconnect with the PSTN (and then are subject to common carriage rules).

"If broadband Internet access service does not afford one of the listed capabilities to users, what effect woudl that have on our statutory analysis?"

It is my opinion that when a provider of broadband Internet does not themselves offer the information service the user is accessing, the provider is not "offering" the information service, but a telecommunications service.

Importantly, if the broadband provider is not responsible to support or engage the user in the use of an information service, they are not offering that service.

This, in context, it may be appropriate to classify some services provided with broadband Internet as information services, but that when not used to access those specific services offered to their users, the broadband provider is providing a telecommunications service, and must abide by those rules.

In Paragraph 28

"We seek comment on how consumers are using broadband Internet access service today"

I se my broadband Internet service only as a transit connection to the Internet. I do not use any service provided by my Internet provider including DNS resolvers, for which I use my own equipment. I find the conclusion that the entirety oof my use constitutes either an information service or a telecommunications service exclusively to be preposterous. Even were I to use my ISPs DNS resolvers, the limit of the information service that I am using is the DNS service itself. My request to retrieve a "tweet" is not serviced by my ISP, but by Twitter.

"Is the consumer capable of accessing these online services without Internet access?"

No, in many cases the services offered by third parties are offered only through the Internet. This question is the analog of "are consumers capable of making phone calls without telephone service" for which the answer is also "No," and which has no bearing on whether the telephone service itself is an information or telecommunications service.

"Could a consumer access these online services using traditional telecommunications services like telephone service or point-to-point special access?"

In principle, this is entirely possible — the information service provided by Twitter, for example, can be accessed via SMS messaging in mobile networks, and nothing except cost and pragmatic considerations prevents private individuals to establish direct communications links to their preferred email, or social networking providers.

"...are we correct that offering Internet service is precisely the service capable..."

You are correct in this assessment. Internet service may be required to access many of the information services consumers use, but this is a technological convenience, not a strict requirement. In the absence of the information service providers, Internet service would not itself provide those capabilities. It is important to understand that ISPs have often offered both information and telecommunications services, as a direct recognition that the pure telecommunications offering is not always sufficient. As more and better services have been made available by companies other than ISPs, the need (and even demand) for those supplementary services has rapidly diminished.

In Paragraph 29

In direct conflict with the statement that "broadband Internet suers do not typically specify 'points' between which and among which information is sent online" is the prevalent and ubiquitous use of URLs (Uniform Resource Locators) which precisely address an online resource (in the same manner that a telephone number is the address of a telephone).

The commission's assertion that "routing decisions are based on the architecture of the network" is true, and is true of telephone networks as well. In land-line networks the telephone number is assigned to a switch serving the user, and that switch is located by examining the telephone number dialed. Number portability ensures that a database is consulted during the routing of the number. Similarly, mobile networks locate the user's device by consulting a database in the network to find the current location of the handset, in order to route the call to the correct destination. This is no different than the process to locate a website the user wishes to contact — location information is retrieved and the network routes the request to the destination, whether by name, telephone number, URI or other means.

Importantly in mobile networks, or when calling an unknown phone number, the user has "no knowledge of the physical location" they are contacting, yet telephony is a telecommunications service.

Consumers do indeed want and pay for this location function the network provides, but these along are not sufficient to substantially change the entire service offered from a telecommunications service to an information service.

Importantly, traditional telephone does not require the user to carefully specify the path through their telephone provider's network, then through the called party's network in order to connect the call; it is unreasonable to assert the same function requires classification as an information service for broadband Internet, when it is present in existing telecommunications networks.

In response to paragraph 30

While providers may use a variety of technologies to transform the representation of user information, interwork networking technologies, or otherwise improve the functioning of their network, these do not constitute a change to the content or form of information the user is sending over the network.

The representation of data can take many shapes. The same speech can be represented by a video recording, solely the audio recording, or a textual transcript. Within computer systems each of these forms of representation may have many different technological encodings and representations. In nearly all cases neither of these (overall form, or technological encoding) will be changed during transmission to the other party, and in many cases the technology used will ensure that any modification is detected (as corrupt/erroneous) and rejected.

The kind of transformations the commission cites are one of several kinds:

"protocol-processing for interworking"

This is analogous to the kinds of processing done in existing telephone networks, especially between national and international networkings in the global telephone network. Telephone service is a telecommunications service and is subject to these transformations, so broadband service is not an information service solely because of these same category of transformations.

"change in content" by use of firewalls

WHile it is certainly a service provided to end users, and also operated for the benefit of the broadband provider (to mitigate attacks on the network itself), firewalls and other systems blocking traffic do not constitute sufficient transformation to mandate the service be classified as an information service. This can be seen as consistent with FCC rules regarding telephone companies being allowed, in fact encouraged, to supply their users with the ability to block unwanted calls (robocalls). Providing such a service does not undermine the nature of the service as a telecommunications service.

As as user of broadband Internet I do not want my provider to alter or restrict my user of the service in any way — and I am happy to release them of any liability as a result.

In response to paragraph 33

In the absence of better and more specific regulation or grants of authority from congress, the framework of Title II is appropriate as it allows the FCC to exercise appropriate discretion while maintaining the necessary protections (common carriage) to preserve the environment of open innovation in the face of the current oligopolies.

In response to paragraph 34

Indeed, the industry around the Internet has been largely unregulated, which has allowed massive societal change, economic growth, and general quality-of-life improvements. However, it should be noted that the innovations have all occurred "at the edge" — it is not ISPs that created YouTube, Facebook, or Netflix, it was not Verizon or AT&T that created Skype. The innovations that we have enjoyed are because the Internet was not constrained in its use, and these are precisely the sort of constraints that I strongly object to — the constraints that block innovation. Common carriage does not block innovation; instead it blocks protectionism. ISPs today do not have a history of innovation; at most they have incrementally improved their networks in order to provide more capacity, both for each individual user, and as demand has grown among all users. This is no small task, and is an important one; but it is infrastructure, the same as electricity, water, or telephone service — many of the concerns that apply to those apply equally well, yet some few may not. This is no reason to throw out the framework entirely.

In response to paragraph 35

I expect my broadband provider to be nothing more than a reasonably-priced transit provider. I do not want them to interfere with or manipulate my traffic through them in any way—if the Title II provisions are not enforced, I have no guarantee of that.

In response to paragraph 36

Whether an ISP offers supplementary services with the offering of a telecommunications service has no bearing on the telecommunications service itself. Telephone service does not become an information service because I subscribe to voicemail—only the voicemail service itself is classified differently.

I do not use, and would not separately pay for any such supplementary services. I am, in fact, likely unaware of the full extent of such services, as I do not have any interest in using them.

In response to paragraph 37

An ISP operating DNS resolvers and caching systems is engaged in both providing the telecommunications service (as an ISP) and some information services (the DNS resolver and caching). The DNS resolver provided is a convenience to their users, though an ISP need not provide this themselves. (Google, among other companies provides a DNS resolver for public use.) The ISP gains from providing a DNS resolver by limiting traffic across their network, as the resolver will cache queries (as part of the operation of DNS) allowing the traffic from their customers to be handled internally, without additional costs to the ISP. Some ISPs will also use their DNS resolvers to return search or advertising to users who attempt to access sites that do not exist. (This often provides little benefit to users, who would be better served by an accurate response that the site does not exist. Instead this provides benefit for the ISP, through advertising revenue.)

Caching is, again, primarily of benefit to the service provider, as it allows them to save infrastructure costs by reducing the required capacity. This may benefit end users in reduced cost of service, but the nature of the service is not vastly different.

Without ISPs provided DNS resolvers, the user's own equipment would need to perform that function. This is not unusual or difficult, and there are providers of DNS resolvers that are not ISPs a user could use as well.

In response to paragraph 39

While I believe the light-touch approach from the past 20 years is appropriate, I dot not agree that the FCC should tie its hands with respect to ensuring that open and unconstrained access to the Internet is preserved, and this specific regulator constraint (that of common carriage) requires the classification as a telecommunications service. It is important the rules are clear, both to industry and to consumers. Classification as a telecommunications service services to provide the necessary tools to provide that clear guidance, and does not require the FCC to stifle innovation or impose heavy-handed regulation—in fact, it is clear the FCC does not intend to do so.

In response to paragraph 40

I do not believe that mutually exclusive categories are the correct framework given the complex nature of modern telecommunications. The difficult to interpret cases of DNS resolvers and caching systems would benefit from a clear separation from the underlying service.

In response to paragraph 42

The peering and exchange of traffic between ISPs is, of course, a private arrangement. However, in as much as a service provider "offers to the public" the exchange of traffic, it is engaged in offering a telecommunications service, and must be subject to the same rules as any other.

This is an especially critical consideration in the case of providers who are "Tier 1"—those that do not pay another provider for transit to any portion of the Internet, but who maintain a peering agreement allowing them to reach every destination. These providers, in particular, have exceptionally powerful roles and as a matter of public interest must be encouraged (or regulated) to abstain from the abuse of that power.

In response to paragraph 48

The Commissions' claim that changing the classification again will alleviate regulatory uncertainty is specious—a single change introduces uncertainty, regardless. Multiple changes only increase, rather than alleviate the uncertainty.

In response to paragraph 50

"Do these isolated examples justify...?"

Yes, absolutely. In all cases the examples have demonstrated that the industry is unwilling to police itself, and that consumers are harmed. The markets are substantially failed, as there is a lack of competition, where competition exists the offerings are often not equivalent, and there are too few for a market to emerge. As well, high costs of entry prevent competition from developing.

"Is there evidence of harm?"

Yes. The ISP industry has record-breaking low customer satisfaction.

In response to paragraph 55

There is not substantive difference between mobile and fixed broadband service in terms of function. I cannot agree with the commission that a service offered to literally millions of end users (the public) is a private service.

In response to paragraph 60

The inconsistency is resolve also by continuing to classify as a telecommunications service, without any need to reconcile.

In response to paragraph 61

Given the vast technological changes happening in the telephony world, i ti snot unreasonable to conclude that the Internet and existing commercial mobile radio services are increasingly and completely interconnected.

In response to paragraph 73

Removing a "vague" set of guidelines will not promote certainty, but instead uncertainty. Firstly, among consumers, who will have to trust and understand what providers are selling individually, instead of relying on a standardized regulatory framework. Secondly, for service providers, removing guidance does not change what consumers expect of them, nor what behavior the commission may find acceptable or not—the set of guidelines increases certainty as it becomes more restrictive. You cannot have light-touch regulation withou significant uncertainty. (Due to ex post enforcement)

In response to paragraph 75

Without any rule, there can be only one outcomes—ISPs continuing to "innovate" by interposing themselves and rent-seeking. Common carriage is the appropriate standard.

In response to paragraph 77

Allowing companies to ignore their customer's protections if they are too small is a mistake; as is requiring burdensome regulation. Enforcement actions should be taken as needed, with a focus on large impact. I see no reason to modulate the protections offered.

In response to paragraph 79

On the surface, yes, an ISP can sufficiently curate and sell a non-telecommunications product. It would be extremely surprising to me if that product would be competitive with the usefulness and expense of un-curated Internet access. I cannot imagine such a service as broadly useful or demanded as the general telecommunications service of Internet access. I certainly would not buy such a service, whatever the price.

In response to paragraph 80

By codifying a no-blocking rule that ISPs already abide by, no burden is applied to the ISPs. The rule cannot be burdensome unless the commissions chooses to make it so.

In response to paragraph 81

The rule is sensible and meets the expectations of consumers buying a telecommunications service.

In response to paragraph 82

Common carriage is the appropriate framework, the reclassification proposed is in error.

In response to paragraph 83

The prohibition on throttling of selected traffic benefits consumers mainly through the innovation that is enabled—an equal playing field enables easy entrance of competitive offerings and new uses that cannot be prediction.

Differentiated prioritization may have specific applications where it provides a higher quality of service, but it must be under the control of the consumer, not the ISP. This is critical to the adequate performance of latency-sensitive applications; but the consumer must be able to choose which traffic is prioritized and how that occurs. The simplest solution is to prohibit ISPs from doing so at all, and require that the consumer's equipment do any prioritization.

Preventing differentiation will increase competitive pressures and is a net positive for consumers.

In response to paragraph 89

As a functioning market requires consumers to have all relevant information available to them, transparency is a requirement.

In response to paragraph 90

The marketplace is not competitive. Transparency and reporting requirements help consumers apply pressure to the few ISPs available to them in order to get the service(s) they want.

In response to paragraph 93

The ideal case is that no exception is allowed to the bright-line rules. ISPs already have safe harbor protection from liability for unlawful content. Giving them the authority to determine such and prevent transmissions of it grants significant privilege to censor the network. "Reasonable network management" is again designed to give ISPs broad latitude to censor the contents of the network. They should be held to the high standards of common carriage. Only specific network management practices should be allowed, but regulation and rule-making can hardly keep up with technology so no other recourse than case-by-case evaluation is possible.

In response to paragraph 94

Protecting the innovation space represented by the over-the-top offerings is extremely important to maintaining the innovation driven by the Internet.

In response to paragraph 95

Mobile and fixed broadband are not different in the service they provide so no fundamental differences should be present in the regulation of the services.

Building a Better Terminal
2015-08-12 10:22

Comments

First, let's start with a bit of background.

I like using a terminal / console. The direct interaction and expressiveness that a command line gives me is very powerful, and efficient. I'm writing this post in vim, using restructured text. I do all my file management in a console window, directly in the shell.

So when I think about what a 'better' terminal would be, I don't mean to replace it with something that fundamentally changes the interaction model. For some, a better terminal would be more like a notebook, with rerunable, editable commands, able to embed images, links, and videos. Interactable with a mouse. That escapes the bounds of what I believe to be generally useful as a replacement for existing terminals. (The IPython Notebook is a very good example of something that is 'like' a terminal, or interactive console, but serves a different purpose and audience.)

All that being said, there are definitely limitations of current terminal emulators that I believe are worth addressing. They fall into a couple of broad categories, with some basic requirements:

  • Robust multilingual / Unicode support
    • Support of the entire range of Unicode codepoints, not just the BMP.
    • Gridded / Monospace rendering, but with the ability to have graphemes span grid spaces (or potentially subdivide them) for legible rendering of different languages mixed together. (Full/half-width katakana, etc.)
    • Normalization of input code points and predictable editing behavior (backspace by grapheme, not code point, input/backspace of combining characters)
    • Configurable font support, including selection of alternate glyphs (important for legible mixing of unified Han ideographs used by multiple languages)
  • Escape sequences / Keyboard shortcut limitations
    • All commands to the terminal must be recognizable without timeouts.
    • All commands must be unambiguous.
    • All commands must leave the terminal in a usable state on failures.
    • Every keyboard sequence of the system must be detectable and usable by the software running on the terminal, they cannot be mistaken for commands to the terminal itself (unless the terminal is specifically configured to use the shortcut, in which case the shortcut must be entirely absorbed by the terminal and not passed to the software running on it.)
    • If a mouse is supported by the input system the terminal is connected to, then the mouse must be either completely unable to interact with the software on the terminal (only with the terminal itself), or the mouse is only able to interact with the software on the terminal, not able to control the terminal itself (when the terminal is focused, in the case of a window manager).
  • Formatting / Fonts / Color / etc.
    • Suport of color, including background color, must be capable of representing a full modern display, 24-bit color, with alpha. (Alpha may not be always useful, but it should be possible, as some compositing window managers can usefully apply it.)
    • Formatting must include normal font variants like oblique, bold, italic, small caps, etc. and not substitute 'bright' colors for these unless desired.
    • Kerning that respects the gridded nature of the terminal may be applied to enhance legibility, especially for symbols or in the case of languages with mandatory ligatures.

Note

These requirements are based on my usage, which is from the perspective of a software developer, with a native language of English, but an interest in correct and pleasant typography, linguistics, and mathematics.

Many of these requirements are born out of the usage of a modern terminal emulator and its environment, given the differences between the current usage and the historical context in which the terminals were developed.

A modern terminal emulator is a piece of software responsible for interacting with the I/O system in some fashion (either through a graphical environment like X, through a kernel primitive like Linux's CONFIG_VT, etc.) and interacting in a specified way with software run on that terminal. It functions as a basic API layer for certain kinds of software.

This is different than the historical usage, which was that a terminal was a device attached to a computer (generally over a serial line) that provided access to the computer, generally a multi-user machine. As such, the I/O system was only able to communicate in-band over a stream of bytes, generally, and many of the limitations of current terminal emulators are a direct result of this arrangement.

Even in the case of a remote terminal, we now have much higher bandwidth in most cases, so a more robust terminal protocol, between software and the I/O system is desirable. Additionally, given no compatibility constraints, we can define an out-of-band mechanism for control of the terminal, rather than needing to rely on a complex and ambiguous stream of bytes for all communications.

I don't have a clear idea of a design, I am only outlining what I feel are the requirements of a 'better' terminal.

Thanks for reading.

6 Word Stories
2012-04-13 18:57

Comments

Consequences

"If..?"

"No."

"Then..."

"Yes."

"Shit."

"Yes."

After the Fall

You missed the Rapture. Good luck.

Alarm will sound...

Nobody's home. Knock, then run fast.

How to boot a Ubiquiti RouterStation Pro from RedBoot over tftp
2010-07-18 14:42

Comments

Just a quick note, more to myself than anyone else, but I wasn't able to easily find this information online, trolling through the OpenWRT and Ubiquiti forums.

I wanted to try booting different kernels on the device before committing to flashing, so that all I had to do was reboot the board and everything would come back up as normal. It seems obvious that this should be possible, and with a bit of digging around in RedBoot docs, trial and error, and a few lucky guesses I was able to figure out the incantation to make RedBoot load an uncompressed linux kernel (ELF) and boot it:

load -m tftp -h 192.168.1.1 openwrt-ar71xx-generic-vmlinux.elf
exec -c "board=UBNT-RSPRO panic=1"

Note that this does not change the root filesystem, so if your modules aren't compatible, this won't work, but I suspect if you use an embedded initrd you can run the system entirely out of RAM without touching flash.

Traditions in Computing
2009-10-13 11:33

Comments

This essay is in reaction to the essay Programmer Superstitions by Jesse Liberty (from 2006)

Technological Traditions

From the outset, our usage of computers and information systems is wholly formed by the designers of those systems. What is, in fact, somewhat remarkable is the unity with which those designs are created, across oceans, cultures, and decades. Some are "obvious," in their correctness, others less so. Many things are done in a particular manner simply because it was the first or simplest way that came to the mind of the designer of the system. Many other aspects of technology are designed to fit the general notions of how that kind of technology — a certain kind of software — is expected to behave. These traditions are mostly nascent and often the learned lore of the industry; few formal instructional programs exist with the intent to teach them.

As technologists, we must be careful to evaluate those traditions in light of the current context; misapplying them can be more damaging than simply ignoring them, as expectations will be set when a particular tradition is invoked, whether by behavior or interface.

At the heart of these traditions are technologists, geeks, hackers, and other highly technical professions — these people have created, embraced, rejected, revised, and updated the various traditions we see today.

Programmer Superstitions

Programmer superstitions are among the most entrenched, whether simply due to a sort of intellectual laziness or something more sinister of these. Programmer Superstitions raises a few points. Most of the points are considered in a vacuum, and in a single-platform, nearly single-language manner. Some examples seem to ignore any technological ecosystem other than the author's native one; this one of the purposes of calling out superstitions, to break out of the universe where we are so entrenched.

Case Sensitivity

I have a gripe with the author's arguments advocating the case-insensitivity of programming languages. He states that the sensitivity of programming languages to the case of the symbols is historical baggage from C that should not be accepted blindly. Not blindly sure, but case-sensitivity was chosen for a reason. At the very least, programmers expect most languages to be case-sensitive — an historical argument to be sure, but the expectation is case-insensitivity. Being an exception is more likely to cause problems.

More importantly, in my mind, is filesystems, command shells, and nearly every other system in a computer is case-sensitive. At least on my computer. Windows filesystems are case-insensitive, but case-preserving, as is HFS on a Mac. This is ostensibly fine, but is tricky to deal with programmatically — program code cannot ignore case in data. The exceptions to default case-sensitivity are, however, very common (e.g. search).

Orthogonality is an important benefit, but could be sacrificed were case-insensitivity a demonstrable win. I have never had a case-sensitive programming language cause undue headache that would have been avoided by ignoring case. Sure, a typos happen, but I am a competent typist and such problems are not so difficult to solve as some may argue. The disadvantage is large: having distinct appearances with the same meaning. Programming languages are one of the most precise ways to express ideas, and the imprecision allowed by case-insensitivity seems additionally burdonsome on an already complex problem.

One other point to make, a social one, is programmers are heavily indoctrinated to preserve data. Throwing away informatin about case is like throwing away the most significant bit of a number. Allowing a variable name, or language keyword to be spelled in a variety of ways allows variations in programming style that can interfere with version control and refactoring tools. Some sort of normalized case is probably used preferentially to any of the combinations of cases allowed. Being able to change nearly every character of a program and have it be functionally identical does not seem a desirable attribute in a programming language.

For more arguments (both ways) on the issue, see this Lambda the Ultimate thread.

Iteration Variables

Another point, that iteration variables in counted loops (like for loops) are often labeled according the mathematical convention of using i, j, k, ..., yet does not make a case for any other convention. He suggests that a and b would "make much more sense" (emphasis added) which is not obvious nor defended. Certainly having an established convention in a related discipline is a sufficient reason? There is no technical reason to call iteration variables any particular name — but many programmers will be familiar with the mathematical custom, especially those with a formal education, so the idiom will be natural and provide meaning where otherwise there would be arbitrariness and individual variation. I don't mean to imply that variables of iteration should always be named in such a mathematical fashion, only when the mathematical analogy is useful. Naming a variable index, count, or any other descriptive name is, of course, ideal in a number of circumstances, especially in deeply nested and complex structures. The simple iteration names are useful for inner loops that have generally uncomplicated structure.

Data Hiding

I do agree with the author that the "object-oriented" principle of data hiding is usually useless and often a noisy and inflexible idiom; though I suspect the author is less concerned with the utility of data hiding than with the discomfort he feels at the lack of it when programming.

The Python language does not have a mechanism to enfore data-hiding, but is a fully capable object-oriented language. (The C# language lacks binary compatibility between properties and object fields, where python does not, so there are technical reasons in some languages for the use of accessors.)

I find I never do things in code that would be useful for the compiler or runtime to have prevented by enforcing data-hiding. Any time I break encapsulation, I leave a big #XXX: hack or similar comment to warn that the design needs to change, or that the code needs refactoring.

Being able to do that (rather than having to do the refactoring at that moment) allows for increased productivity, at the expense of some overhead when the refactoring needs to be done en masse.

Content Overhaul
2009-08-06 17:50

Comments

This site has undergone some major overhauling to get to the (new) current state. The code is now completely custom, using the filesystem as a content backend (so I can use Mercurial to version content).

The design is simpler, and more focused on the content, though I imagine I will be refining some elements over time; though, I certainly couldn't have designed anything like this myself. I give tremendous credit to Rodrigo Galindez for making the design and providing it for free use.

That being said, there are some quirks to the design, especially at screen resolutions smaller than 1024x768 and with WebKit.

Most of the same content that was here is still here, hopefully under the same URLs, though a few are different. Though there are a number of articles that were imported, and imported again, and so on from some time ago that may have broken text or links. I apologize, and will be working through correcting them as I have time. Along with fixing bugs in the code. If you see a 500 error on the site, please email me, it's not something that is someone else's fault, it's my code, I'd like to know when it breaks.

WRT54G and OpenEmbedded
2009-07-24 17:00

Comments

Some time ago I bought a cheap `Linksys WRT54G`_ because I aquired a laptop with WiFi. It worked well enough, but being the technologist I am, I soon found out that the device ran an embedded Linux distribution, and that it was possible to replace the firmware with a more capable (if more arcane) variant. I did so, using OpenWRT_. I was quite satisfied with this distro, though wireless was flaky due to requiring the use of a proprietary 2.4 kernel-only Broadcomm driver for the wireless. To make a boring story as short as possible, I'm now using a separate device as an access point and using the WRT54G solely as a router.

Thus, I take my first stab at using a 2.6 kernel on the device, upgrading to a recent build of OpenWRT_. Unfortunately, I don't like what the project has done recently, and want less magic configuration. UCI seems like a nice system, but I just want what I'm used to when configuring a Linux system, a bunch of files in /etc. I don't need a unified syntax, I already know the tools and the configuration syntax for the functionality I need. I don't want to learn yet another specialized language for doing the same damn thing.

Don't get me wrong, I understand the need for UCI, I just don't have the same needs. (Dual-stack IPv6/v4 with firewalling and VPN, DNS and static DHCP leases, to start...) So I'm left with radically modifying the OpenWRT build system, or rolling my own.

So I turned to the OpenEmbedded_ project, which provides a customizable and flexible cross-build system for many targets, including the WRT54G.

Setting up OpenEmbedded

First things first, in order to build an image for the WRT54G, you need the OpenEmbedded_ build environment set up. Follow the `Getting Started`_ guide and come back when you're done.

Then, create a local.conf file with the appropriate values for the WRT54G:

BBFILES = "/stuff/openembedded/recipes/*/*.bb"
DISTRO = "wrt54oe"
MACHINE = "wrt54"

PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc-initial:gcc-cross-initial"
PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}gcc:gcc-cross"
PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}g++:gcc-cross"
PREFERRED_PROVIDERS += " virtual/${TARGET_PREFIX}binutils:binutils-cross"

Just to make sure everything is working, try building task-base:

$ cd /stuff/build/
$ bitbake task-base

This will take some time, so feel free to go do something else for a while.

I had this part fail several times, needing manual downloads, md5 corrections, etc. Your milage may vary.

Docutils System Messages

System Message: ERROR/3 (<string>, line 1); backlink

Unknown target name: "linksys wrt54g".

System Message: ERROR/3 (<string>, line 1); backlink

Unknown target name: "openwrt".

System Message: ERROR/3 (<string>, line 10); backlink

Unknown target name: "openwrt".

System Message: ERROR/3 (<string>, line 23); backlink

Unknown target name: "openembedded".

System Message: ERROR/3 (<string>, line 29); backlink

Unknown target name: "openembedded".

System Message: ERROR/3 (<string>, line 29); backlink

Unknown target name: "getting started".

google-voice-call.py
2009-04-30 17:29

Comments

Simple python script that automates the Google Voice web interface to make outgoing calls. Requires httplib2, simplejson, and BeautifulSoup

This script is not tested on Windows but should work, it looks for a file in the users home directory named .gvoice_auth to determine the credentials to use and the default number to use for the 'from' leg of the call. The file is a JSON encoded object with the following attributes:

user
Your username / email address that you use to login to Google Voice
passwd
The password you use to login to Google Voice
default_from : optional
The number to use for your leg of the call if you don't specify a different one.

For example:

{'user':'me@gmail.com',
 'passwd':'lamepasswd',
 'default_from':'+15855551234'}

Then, simply run the script:

$ python google-voice-call.py 5855551235
Call request returned:  {"ok":true,"data":{"code":0}}

Here's the code, though you probably want to grab google-voice-call.py directly, cut and paste can be annoying.

#!/usr/bin/env python
'''
    A ugly script to pretend that Google Voice has an API for calling out.
'''

import sys, os, re
import urllib, httplib2
from os import path
from simplejson import load
from BeautifulSoup import BeautifulSoup

def main():
    if not path.exists(path.join(
                        path.expanduser('~'),
                        '.gvoice_auth')):
        return """
You need to create a config file in your home directory\
called '.gvoice_auth' with your Google Account's credentials.\
"""
    else:
        creds = load(open(path.join(path.expanduser('~'),'.gvoice_auth')))

    if len(sys.argv) == 2:
        fromnum = creds['default_from']
        tonum = sys.argv[1]
    elif len(sys.argv) == 3:
        fromnum = sys.argv[2]
        tonum = sys.argv[1]
    else:
        return "Invalid number of arguments\nUsage: %s <number> from_number]" % sys.argv[0]

    auth_tok = authenticate(creds['user'], creds['passwd'], 'grandcentral')


    r = call(tonum, fromnum, auth_tok)

    print "Call request returned: ", r


def call(number, fromnum, auth):
    data = {
        'outgoingNumber': number,
        'forwardingNumber': fromnum,
        'remember': 0,
        }

    main_uri = "https://www.google.com/voice"
    call_uri = "https://www.google.com/voice/call/connect/"
    headers = { 'Content-Type': 'application/x-www-form-urlencoded',
                'Cookie': 'gv=%s' % auth }

    h = httplib2.Http()

    response, content = h.request(main_uri, 'GET', headers=headers)
    if not response['status'].startswith('2'):
        raise Exception(response['status'] + ': ' + content)

    soup = BeautifulSoup(content)
    _rnr_se = soup.find('input', attrs={'name':'_rnr_se'})

    data['_rnr_se'] = _rnr_se['value']

    response, content = h.request(call_uri, 'POST', body=urllib.urlencode(data),
                                  headers=headers)
    if response['status'].startswith('2'):
        return content
    else:
        raise Exception(response['status'] + ': ' + content)


def authenticate(user, passwd, service):
    '''
        Basic perform ClientLogin method of authenticating a Google Account.

        Returns the Auth=... bit required in the Authorization header.
    '''
    h = httplib2.Http()
    uri = 'https://www.google.com/accounts/ClientLogin'

    headers = {'Content-Type': 'application/x-www-form-urlencoded'}
    request = urllib.urlencode({
        'Email': user,
        'Passwd': passwd,
        'service': service,
        })

    response, content = h.request(uri, 'POST', body=request, headers=headers)
    if response['status'] == '200':
        return re.search('Auth=(\S*)', content).group(1).strip()
    else:
        raise Exception(response['status'] + ': ' + content)

if __name__=='__main__':
    sys.exit(main())

UPnP ATSC Live Video Streaming
2009-04-30 16:15

Comments

This article will guide you through the process of setting up live video streaming from a DVB (ATSC) capture card in Linux to a DLNA UPnP Media Renderer that supports MPEG-2 Transport Streams (such as a PS3).

What you need

Mediatomb
Mediatomb is a DLNA/UPnP media server for Linux. You'll want to set up mediatomb to play regular files to your MediaRenderer before attempting the configuration in this guide. It helps to know how the javascript import and transcoding configuration in Mediatomb works, but this guide should be adequate as long as you don't need to do anything fancy.
DVBStreamer
This is the tool that directly talks to your DVB capture card and streams the video. This would be all that you need if your MediaRenderer understands how to listen to a UDP stream. The can't do this (or I couldn't figure out how to do this) so we need one more tool...
NetCat
The swiss-army knife of networking, this tool allows us to take the UDP stream that DVBStreamer gets us and pipe it through a FIFO with a simple script.

Configuring Mediatomb

The tricky bit to get streaming working correctly is getting Mediatomb to do the hard work of talking to the DLNA devices — in a language they understand.

You'll want to set up Mediatomb and get it talking to your UPnP media renderer before trying to do this; you can try to do it all at once, but don't say I didn't warn you.

Firstly, you need to use an import script rather than the built-in import provided by mediatomb for scanning files. The XML config for this will look something like the following:

<scripting script-charset="UTF-8">
    <common-script>/usr/share/mediatomb/js/common.js</common-script>
    <playlist-script>/usr/share/mediatomb/js/playlists.js</playlist-script>
    <virtual-layout type="js">
      <import-script>/etc/mediatomb/import.js</import-script>
    </virtual-layout>
</scripting>

This config goes in the "<import>" tag of the "config.xml" file.

You also need to add a map from a file extension to a mimetype, I used ".live" -> "video/x-dvb-stream", but you could use whatever you like, the mimetype doesn't escape mediatomb, it's just used so we can tell mediatomb to "transcode" the files into something else. Add the following to the "<extension-mimetype>" tag:

<map from="live" to="video/x-dvb-stream"/>

Finally, you need to add a transcoding profile for the "video/x-dvb-stream" mimetype to transcode it into a "video/mpeg" stream. The "<transcoding>" tag should look something like the following:

<transcoding enabled="yes">
  <mimetype-profile-mappings>
      <transcode mimetype="video/x-dvb-stream" using="dvb-stream"/>
  </mimetype-profile-mappings>
  <profiles>
      <profile name="dvb-stream" enabled="yes" type="external">
          <mimetype>video/mpeg</mimetype>
          <accept-url>no</accept-url>
          <first-resource>yes</first-resource>
          <hide-original-resource>yes</hide-original-resource>
          <agent command="/path/to/bin/dvbstream-live" arguments="%in %out"/>
          <buffer size="524288" fill-size="262144" chunk-size="1024"/>
      </profile>
  </profiles>
</transcoding>

Don't worry about the "dvbstream-live" script just yet, we need to get DVBStreamer set up first.

The final piece to the puzzle is the "import.js" script. This is where having some knowledge of javascript and what you want to do helps quite a bit. You should copy the default import.js script from wherever it's installed, and then add the following code:

function videoIsLiveChannel(obj) {
    var pattern = /(.*)\.live/;
    var match = pattern.exec(obj.title);

    if( match ) {
        var container_chain = new Array('video', 'television' );

        obj.title = match[1];

        print("Parsed channel info: " + obj.title);
        addCdsObject(obj, createContainerChain(container_chain));
        return true;
    } else {
        return false;
    }
}

You also need to add the following to the top of the "addVideo" function:

if(videoIsLiveChannel(obj)) {
    return;
}

This tells the import script that if the file to be imported is a ".live" file, we want to add it to the "video/television" section of the UPnP media library, sans ".live" ending. Strictly speaking, adding this to the import script is not necessary, but it makes the listings look a bit nicer.

Setting up DVBStreamer

DVBStreamer is a simple tool that reads from a DVB capture card and can stream (and filter) the broadcast to a "udp://" url. To get DVBStreamer up and running you will need to already have a "channels.conf" file, generated by a program like "dvbscan". Then simply run the "setupdvbstreamer" command with the appropriate options to get dvbstreamer working.

Use the DVBStreamer Wiki to look up how to do some basic tests, stream some video to a file, or use mplayer to watch a UDP stream; essentially, you want to verify that DVBStreamer is working correctly and doing what you expect.

To stream using the scripts below you need to run DVBStreamer as a daemon and set to stream to "udp://localhost:12346". Create a file "dvbstreamer.startup" with a single line:

System Message: ERROR/3 (<string>, line 145)

Cannot find pygments lexer for language "none"

.. sourcecode:: none

    setmrl udp://localhost:12346

Start DVBStreamer like this:

dvbstreamer -f dvbstreamer.startup -d

Now all you need to do is get the scripts to do the "transcoding" and create some ".live" files for your television channels.

The "transcoding" script(s)

There are two scripts that are used by the streaming system, one changes the channel in DVBStreamer so the correct video is streamed, the other pushes the stream from the UDP stream to a file (FIFO) which mediatomb provides as an argument.

Here's the first, the channel changing script. It requires python >=2.5 (and <3.0). It expects the path to a ".live" file as it's first (and only) argument, and connects to the DVBStreamer instance (using the default authentication credentials) to select the DVBStreamer service corresponding to the name of the file. e.g. "/home/media/21-1 WXXI-HD.live" would select the "21-1 WXXI-HD" DVBStreamer service.

#!/usr/bin/env python
from __future__ import with_statement

import sys
import socket
from contextlib import closing
from os import path

class DVBStreamerException(Exception):
    pass

def get_response(streamer):
    resp = []
    while True:
        try:
            resp.append(streamer.recv(4096))
        except socket.error, e:
            if resp and e.args[0] == 11: # temp unavail
                break
            elif e.args[0] == 11:
                pass
            else:
                raise
    if not resp:
        raise DVBStreamerException("Connection closed?")
    resp = ''.join(resp).splitlines()
    if  not resp[-1].startswith('DVBStreamer'):
        return resp + get_response(streamer)

    code = resp[-1].rsplit('/', 1)[-1]
    if int(code.split()[0]) != 0:
        raise DVBStreamerException(resp)
    else:
        return resp[:-1]

def dvbstreamer_cmd(cmd, auth=False, adapter=0):
    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
        sock.connect(('localhost', 54197+adapter))
        sock.setblocking(0)
        #print "connected, waiting for ready...",
        get_response(sock)
        #print 'got ready: '
        if auth:
            #print 'sending auth...',
            sock.send("auth dvbstreamer control\n")
            get_response(sock)
            #print 'auth success'
        print 'sending cmd: ', cmd.strip()
        sock.send(cmd)
        return get_response(sock)

def main():
    #l = open('/tmp/channel.log', 'w')
    #print >>l, sys.argv
    channel = sys.argv[1]
    if channel.endswith(".live"):
        channel = path.splitext(path.basename(channel))[0]
    #print >>l, channel
    dvbstreamer_cmd("select %s\n" % channel, auth=True)

    #l.close()

if __name__=='__main__':
    main()

Here's the (much simpler) script that calls the above script and hands off to netcat to push the stream to the FIFO.

#!/bin/bash
/path/to/dvbstream-channel "$1"
exec nc -l -u -p 12346 localhost > "$2"

Put these two scripts someplace you can remember and ensure they can be executed by the user running mediatomb.

Wrap-Up

That should be all there is to it, you can download a tarball containing the scripts and config snippets in the article to make life easier.

Random Panic Linux Kernel Module
2009-04-20 00:00

Comments

This kernel module, entirely useless, and probably not even very funny, is meant to torment people you don't like. And who let you have root access anyway.

As the description states, when loaded it will set a timer for a random amount of time into the future, 0-65535 seconds, to call panic() with a randomly selected message (grep'ed from the kernel source tree, see panic_msgs.h)

This really should serve to frustrate and annoy anyone attempting to use the machine if the module were loaded automatically on boot. Don't do it without expecting a nosebleed when they find out it was you.

The source can be downloaded from the mercurial repository or you can download a tarball.

Not much to it, type make and it'll compile. insmod it and it'll do it's thing.