Marketplaces Enforce Master-of-None Mentality

Marketplaces are great. On my Android phone I have, at my fingertips, a huge amount of applications that just work. Marketplaces provide us with a sense of security. To uninstall the app, there is guaranteed to be exactly one thing you must do. To install an app, there is exactly one way to install it. It is self contained, there are no dependencies I have to install. Configuration is non-existent, if at all. Discovering how to launch your app is straight forward. It just works.

Let's contrast that with a typical Linux system. I use Arch Linux. So, when I go to install an application, I use pacman -S someapp. And I cross my fingers and pray that it works. Usually it does. Sometimes I have to manually download and install things that aren't in this blessed "marketplace" of sorts. It's never as seemless as "closed" markets though. A linux application can do anything. It could corrupt my system(if I give it sudo), it could trash my home directory, it could install spam that I could never figure out how to uninstall.

These are two sides of a coin. They are naturally at ends. There isn't really a good way of curing these problems with Linux. Most people would say they aren't problems, but rather design choices(myself included).

marketplace

Dependencies... how I miss thee

So, what's this all about? If you look on the Android Marketplace, iOS AppStore, or god forbid the Windows Store, you'll see a stark difference compared to Arch Linux's packages. And no, it's not the open source aspect.

If you want to search through a file in Linux, you'll probably use something like

cat somefile | grep 'something'

you'll use the cat utility to read the file in and pipe the contents to grep, where grep will search across the file for "something".

How do you do that on Android? Or Windows 8/RT?

Basically, you can't. At least, not in a good way. With Android, file managers is possible, and most of them include some basic searching capabilities, but you won't get the power of grep. You won't be able to do awesome shit like you can by combining the strengths of different applications.

If I wanted to write a file search utility for Android, I'd have to first build a sub-par file browser to navigate to the file, and then implement my actual search functionality.

Markets enforce master-of-none mentality

I once had a magnificent plan to port my scripting language to Android. How much work would that require?

  1. File browsing/saving/loading
  2. Text editor (syntax highlighting, searching, etc. More than just a text box)
  3. My programming language

And that's just the start. If I want to provide APIs in my language to search in files, I have to implement that. If I want network access, I have to provide that. There is no netcat, or grep that people could utilize instead of my sub-par APIs.

Why netcat doesn't exist in markets

If you wanted to implement a netcat utility in any marketplace, it'd be fairly pointless. The power of netcat comes from being able to pipe it to other places that the original authors never even dreamed of. What's that, you want to make a TCP/IP proxy?

nc -l -p 8080 | nc example.com 80

You want something that can encrypt a file and send it off somewhere?

openssl aes-256-cbc -salt -e < file-to-transfer | nc example.com 9999

How would you do this in a marketplace application? Sure, maybe you could cobble together some solution like finding a dedicated TCP proxy. And then finding a file encrypter and a TCP/IP program that can send files... but this requires that someone developed such an application beforehand.

You can't just create some general purpose utility. You must create some "multi" purpose utility where you came up with all of the interesting use cases you could and implement them. If you missed one, then there just isn't a solution to that problem. There is no way to combine your program and some other program to solve the problem. It's all or nothing.

It's not just markets

If you notice, desktop Windows does this to a certain extent as well. It's I/O redirection is downright terrible. (although I hear Powershell is nice) This is probably why you see all-in-one applications everywhere. Linux has a general "air" about it that encourages you to make things modular and enable the utilization of other tools where possible.

However, marketplaces is the only place where this is actually enforced. Windows 8 has extremely limited IPC functions. Oh, you gave me a (very limited) search API that works across every application, big whoop. Windows 8 especially enforces it. Did you know that you can't make a general purpose text editor in Windows 8? Impossible. There is no way to open every file with a single application. They enforce you to declare which file extensions you'll be allowed to edit (and no, * doesn't work).

Finally, the bugs

Have you ever encountered a bug in a walled-garden application? Of course you have. Would you say you encounter them more than on desktop application? Probably. Developers can't worry about only one thing because if they don't implement it, then their application can't do it. You get a feature request in your netcat-want-to-be for sending text on-demand instead of files. Now you have to implement some kind of text editor. Now some people want to be able to return an automated response that returns the current date and time. Yea, good luck with keeping up with the wishes of your users.

Developers can't just worry about the one thing they do good. They also have to worry about all the things people might want to combine to make your application more useful. This is why I believe that most market applications have more bugs than their counterparts in desktop operating systems.

For the picky

Yes, I know I probably have some false assumptions, but I'm not far off. I'm no pro in Android and such. It's probably possible to do some rudimentary IPC and maybe even some kind of dependency stuff... but it's not the norm, and I know it's probably not easy for you OR the end user.

Posted: 4/30/2013 4:22:51 AM

My first DDoS attack, kinda

Enter my home town: Wilburton, Oklahoma. A small town(~3000 people) you've probably never heard of, and probably won't again after this article. When you think Oklahoma you probably think back woods and redneck, not computers. You're usually correct, but I didn't quite fit that stereotype. This is how I came to be banned from using the computers at my high school for a semester.

How I got here

I figured out what programming was around 13 (7th grade). Instantly fell in love with it. So much so that I'd stay up til 3 AM on school nights learning more about it, leading to failing grades until some parental intervention stepped in.

So, I knew my way around a computer. I was young. Just the right kind of person to be a bit dangerous. Luckily I never enjoyed the hacking scene and never crossed over to the script kiddie stuff, but I knew the basics of vulnerabilities.

The day the internet broke

I was a junior the year this happened. It was sometime in the first semester. There is usually an inherit trait among programmers: curiosity. A yearning for wanting to know the consequences of a previously untried action. My lunch periods usually consisted of boredom. Recently the blocks at the school had been relaxed, so flash games could be played in the library. Hence, that was where me and my friends went to during our free time. Someone mentioned something about the command line and hacking. I don't quite remember what led up to it, but I ended up typing something like this:

ping -n 10000 -l 10000 1.2.3.4

The IP(1.2.3.4 is just a placeholder)) I used was the particular IP returned when content was blocked. They did block some content, but it wasn't horrible yet(at some point my own website got blocked for flash games. Ruthless!)

For the non-technical people reading: This is a command which basically says "send this huge message to a server and tell it to send back a huge reply". This command took me a tiny bit of research (using ping /?) to even know. I wasn't a black-hat by any means.

So, the technically minded out there are thinking "there is no way this would break anything"... Well, it didn't.. but then I ran it from 4 other consoles on the computer. At this point I started to hear "hey is the internet slow for you?" asked around the room. This is where I made the naive mistake of running with it. So, I opened like 10 command lines on this computer running this crude flood ping. Then, I went to the free computer beside me and did the same thing there. I think I did it on a total of 4 computers.

And then everything stopped working

I did not expect what came next. I had expected for there to be fairly strict controls on bandwidth. I thought I just maxed out the router in the library. Like the young naive teenager I was though, I left it running... on all 4 computers.

I was in 5th period when I started to realize what I had actually done. The teacher tried to load something on the internet and it wasn't working. "Connection timed out". I would later find out that I had brought the networks of the high school, middle school, elementary, and portions of the college to a stand still; as well as inflicting some damage on the IP address I used

About 30 minutes later it started working... another 5 minutes later, just before class was over, the principal knocks on the class' door. "I need to speak with Jordan".

"Ah shit."

So, we meet in the principal's office. My friend Joe is also there, and the IT guy. He starts saying that someone maliciously attacked the school network and their upstream provider for censorship. My friend Joe stares blankly and says "I have no idea what you're talking about." At this point, I know there is no point in not confessing, so I do. "Yea, he didn't have anything to do with it. I did it".

I can't remember of what follows but I was scared out of my mind. Not of police being called or whatever punishment they were going to arrange. The only thing on my mind was how much trouble I would be in when I got home. I got in-school-suspension before for some stupid thing. I was assigned 3 days of it... That was the worst month of my life though at home.

Anyway, so he sends me back to class and says he'll decide my punishment. Later he calls me back in and hands me my sentence "You can not touch a computer for the rest of the semester". Luckily, I didn't have a programming or typing course. I had both the next year.

Somehow, I manage to never have to tell my parents. They didn't find out until I was moved out.

It's hilarious looking back on it, but man was it scary when it was happening.

Unforeseen Consequences

This lead to interesting situations. The IT guy was usually not on premises, so I would literally get called out of classes to fix some teachers' computers. During this time, I had to walk them through what to do, since I couldn't touch the keyboard or mouse. That was annoying, at best.

The most awesome part of this story is that I landed my first programming job literally, without a doubt because of my fame from crashing the network. My first boss had a son who was a grade below me. He was complaining about how he can't find anyone who knows anything about computers in this small town. His son popped off that I do. This conversation took place in the drive through of McDonald's. I happened to be working drive through. So, I say "that'll be ..." and he proceeds to ask "Hey what programming languages do you know". Completely caught off guard I start saying the last language I used. "C++". He gave me his business card and we exchanged numbers. About 2 months later I graduated from high school and began working there gaining vital experience right out of the gate.

I had quite a bit of fun with other stuff in high school as well.

Other "hacks"

CD_Opener

I wrote a program called CD_Opener. It was my first real program to use threading and the Win32 API at the same time. It was a simple and stupid program. It did nothing but open a pop-up window with just an "OK" button. In one thread it would keep your CD drive open no matter what, in the other it told a story through these popup windows you clicked through. There were two versions, titled "ending" and "nonending" The nonending version told the story so that it fell into an infinite recursion along the lines of "and a dialog box opened on my comptuer... and do you know what it said?". There was no way to kill it other than to use task manager. The ending version was more polite and eventually the story did come to an end

After showing my friends this program(and how to run it), it was not uncommon to come in to the library to see 6 computers with their CD drives stuck open and a familiar popup dialog.

Getting around censors

At one point I built a small PHP script to dumbly download whatever files I told it to and give me a link to it hosted by my server. This was only effective though for flash games and other single-file things.

The unfinished senior prank

I found an absolutely brilliant vulnerability in the high school network my senior year. Basically, all the computers used a common "student" account. The student account was of course just a template. When a student logged in, it copied over the template. Changes in this way didn't persist.

However, the huge flaw with this design was that I found I way to put my own files into the template student account. I tested it with a small batch file and logged in with another computer and indeed, it did run on startup. I then proceeded to delete the batch file. I thought long and hard about writing a senior prank program that would run on almost every computer on the school at a certain time of day. Nothing harmful or distasteful, but not something one would forget either.

I ended up writing a stub program, but never finished it and as far as I can remember never exploited this vulnerability. I vaguely remember leaving a text file in some obscure folder, but I probably ended up deleting it after a while.

Many more vulnerabilities lie here, but I won't go into them here. Our IT guy didn't have the best knowledge of basic security.

Conclusion/Disclaimer

If you're under 18 and reading this, Please don't try to break your school's network. There are many more positive ways to get a reputation. My school was easy on me. I have read(as an adult) about similar (trivial) cases of such things where the school involves the police and the kid receives a record that'll stay with him for life.

Posted: 4/18/2013 3:45:49 AM

SD cards on Mbed

So, I don't understand why, but apparently hooking up SD cards to an Mbed isn't just "plug in these wires and import this library"

The official library OR something in my circuitry seems to have a bug. So that I won't forget how it all works though here is my quick write up:

First off, I'm using the new Sparkfun SD card breakout board, so I've "translated" the pinouts between SPI and SD formats:

**NEW** SparkFun SD Breakout Board
MicroSD Breakout    mbed
   D3  o-------------o 20    (DigitalOut cs)
   CMD  o-------------o 11    (SPI mosi)
   VCC o-------------o VOUT
   CLK o-------------o 13    (SPI sclk)
   GND o-------------o GND  
   D0  o-------------o 12    (SPI miso)
   CD  o
   D1  o
   D2  o

Then, for some reason, the SD card library for mbed would never work the first time I ran it. So, I had to fork it and make it so that "if initialization fails, try one more time, just in case". It's delightfully horrible, but I'm tired of trying to find the correct solution to the problem

That forked repo is located at SDFileSystem_tryagain

After doing all this, it finally "just works"

Notes about my setup:

  • The try again bit could be caused by noise. the SD card is about 2-3 inches from the mbed and requires long lines
  • I'm using a 4G SDHC card produced by SanDisk
  • I have no idea what I'm doing with hardware :)

Hopefully this helps some helpless soul searching on google. I know I had a ton of problems finding any reference to this problem, even though people have told me that it does happen

Posted: 4/16/2013 5:37:39 AM

Networking Terms In Plain English

Extremely simplified. Do not assume these are perfect definitions. This is what I would use to describe a network to my parents or someone else that is only concerned about consumer use.

The goal of this is so that I can explain something to you, and you can at least get the gist of what I'm saying without me having to explain each and every technical term.

ISP

Internet Service Provider. AT&T, Comcast, etc. Usually provided by

  • DSL
  • Cable
  • Dial-up
  • Satellite
  • Cell towers(3G/4G/LTE)
  • Fiber

Infrastructure

This usually refers to your ISP's infrastructure from a location they own to your home. If you have DSL and a phone line got cut on the way to your house, you could say that's an infrastructure problem, kinda.

Public IP Address

This is basically your network's "address" to the world. Most ISPs provide you with one IP address for your network, though it's possible to have more than one.

TCP/IP/UDP Ports

A port is basically a "channel" that communications happen between two IP addresses. If your IP address is your "Address" on the internet, the port number they use is the "PO Box".

LAN

Local Area Network. This is your local network. The public internet can not see this network unless you explicitly share it with them.

WAN

Wide Area Network. This is the internet. A "WAN port" is a port which connects directly to the internet (ie, to your ISP's routers and other equipment)

Modem

A modem is a device which takes an encoded connection from an upstream provider(your ISP) and decodes it so you can easily communicate with it from your standard ethernet network. Most modern modems have built in routers so that you can have an "all-in-one" device that creates a usable network.

Router

A router is a lot of thing. It's primary purpose is to share your public IP address among more than one device with a method called NAT. Most routers also have a built in switch so that you can easily hook up more than one computer to the router, though technically a router could function with just a WAN and a LAN port. Routers usually handle NAT and firewalls.

NAT

NAT stands for Network Address Translation. This is the process used by routers to take your 1 public IP address and let as many computers as you want to use the internet behind it. Without NAT, you'd be limited to one device using the internet at a time without paying for more public IP addresses

Switch/Hub

A switch is basically the same as a hub, as far as you are probably concerned. A switch works as a "repeater" so that you can connect multiple devices to a single router. Without switches, your router could only connect to 1 computer. A switch is not a router. A router handles NAT and firewalls to allow you to share a single connection among different computers. A switch just makes it so that multiple computers can "connect" to that single core connection

Firewall

A firewall WILL NOT protect your computer from viruses, at least not with modern networks. A firewall prevents the internet from touching your private network(LAN). With NAT, a firewall is required because of how NAT works.

Port forwarding

Port forwarding is the process by which you selectively allow a certain device on your network to be reached from the internet. This is basically making a "pinhole" in your Firewall to allow the internet to go to a certain device using a certain port.

Wifi

This is a wireless technology which can replace traditional ethernet cables. If you have a modem and router(without wifi), to enable wifi on your network you must buy a wireless switch. It's just like a switch(lets multiple computers connect to your single connection), except for it's wireless instead of wired

802.11b

This is the early verion of Wifi. It's slow, but not usually slower than your internet connection. (it's usually not the bottleneck)

802.11g

This is not the newest version of Wifi, but it's not bad. It's fairly fast and it will be fairly rare that it is slower than your internet connection.

802.11n

This is the newest standard and is blazing fast. If you have an internet connection that is faster than this protocol allows, you probably don't need to be taught these terms

Wireless Speeds

This isn't a term, but wireless speed usually is limited by either your connecting device(ie, smartphone, laptop, etc) or your wireless modem/router/switch. If your have an 802.11n wireless switch, but your smartphone only supports 802.11g, they can still talk to each other, but it won't be at 802.11n speeds.

NAS

Network Attached Storage. This is a device such as a harddrive that is connected to your private network(LAN). This allows you to access this harddrive from any device on your network. These make great backup systems. If you have one of these and use Wifi, you'll want to use 802.11n when possible

DNS Server

Domain Name System Server. This is the server which looks up "names" on the internet. For instance, you type "google.com" in your browser, the internet doesn't know where "google.com" is, it only understands IP addresses. So, it asks a DNS server "who the hell is google.com?" and the DNS server replies with "here's google's IP address"

IPv4

This is the "old" IP address system. There are less than 4 billion addresses available, and we are approaching that limit. As such, IP addresses are getting scarce.

IPv6

This is a huge topic, but basically all you need to know is it's the "new" IP address system. We are currently running out of IPv4 addresses because there are less than 4 billion available for use. We obviously are approaching that many devices on the internet and as such, they are becoming scarce. IPv6 increases this number so that you can have multiple public IP addresses to your network. IPv6 and IPv4 can't really "talk" to each other though. If you have an IPv6-only device, it can't talk to a website served using only IPv4.

It's the future, but it's not here yet, so it's best to have both IPv4 and IPv6 support at this point.

Internet Backbone

This is the "core" of the internet and consists of very high capacity routers owned by powerful companies. The backbone of the internet is provided by (for the most part) very fast fiber

Hopefully, you know enough to keep up moderately (get the "gist" of) when someone explains something about your network/internet now.

Posted: 4/12/2013 2:43:43 AM

LightVM + MbedConsole = Not Dead Yet!

So, I've recently been wanting to really get MbedConsole to a all-in-one system, complete with a programming environment. After spending a few months shifting around different ideas on the best way to implement a programming language in such a small amount of resources, I've decided to go another route.

Yesterday, I created a new bitbucket repo called LightVM. Here, I will implement a very lightweight VM complete with a self-hosted assembler with bootstrapping. After getting it to run good on my PC I'll port it to the mbed and eventually also see if it'll work on an ATMega16 or some such.

So, what all will be added to MbedConsole?

  1. LightVM implementation complete with system calls
  2. Assembler for LightVM which runs in LightVM
  3. A very basic file editor. It'll probably be as horrible as ed
  4. "real" filesystem access.
  5. Because the semi-hosted filesystem sucks horribly, I'll be trying to add SD card support

So again, not dead! Check out LightVM. When it gets to a usable state, I'll start working on the bootstrapping assembler and then the actual assembler. The file editor will probably not be made in LightVM for performance and cost of implementation reasons.

Posted: 4/5/2013 12:16:46 AM

A Proposal For Spam-Free Writeable APIs

I've been having an interest in Bitcoin recently, but it would appear I'm too late to the party to make any money on mining. So, what's the next best thing? Taking their idea and using it elsewhere.

The idea behind Bitcoin is to make a particular thing a rare commodity. Now let's pretend we have a website like say http://stackoverflow.com We want to make a public API for it that is writable. Current options appear to be

  1. API keys which require a human to register
  2. ????

I'll throw a second option into the mix. "API Coins" which require a fair bit of computing power to create and are only good in a certain context.

Let's say you wanted to make an account at stackoverflow with a machine that didn't require any human interaction, or rather, didn't require a captcha, valid email, personal info, etc. In theory, a program could register it completely in an automated fashion.

My proposal to prevent masses of spam bots: make it expensive. Use a bitcoin like scheme. Instead of SHA256, I'd go for scrypt because it's so mostly better on CPUs rather than GPUs, and thus capable of executing from Javascript.

So, when you visit the register page I provide something like

  1. Conditions a hash must match (difficultly)
  2. The value hashed must contain a certain provided phrase (to prevent pre-mining of API coins)
  3. That's it!

You calculate a hash which matches and poof! You've got an API key. Ideally, this would be a process that would take no more than 5 minutes on the slowest of hardware. Now, when you need to perform an operation, there will be another hash request, but it won't be as intense as the creation of your API key... but if you're a bad boy, your API key will get banned and you'll have to generate a new one.

Now, how does our site know that API keys are "valid" without pre-mining risk? The key is to make the nonce phrase be random and unique, but slightly persistent. So, when the request is made to get the nonce, it is stored for say an hour. If the API key isn't "found" within an hour or two, it's considered invalid. This would prevent batching of API key creation.

To help to enforce these "hard" checkpoints, if a user, say wanted to post a comment, they'd be given a request like the API key request. A certain difficulty and a phrase to be contained within the pre-hash value. Ideally, this would be significantly easier than generation of an API key.. You could also enforce throttling at this phase by increasing the difficulty for their account as they post more and more things.

The other awesome part about this scheme? It's anonymous other than the IP address in the logs. You can be reasonably sure that it's a human posting while getting absolutely no personal information and storing absolutely no personal information. No passwords needed. You effectively have a sort of private key instead, stored in a cookie or some such.

This also enables awesomely easy registration for users of your API users. "What's an API key?" crops up plenty. Eliminate the need for it!

Some unsolved problems with this approach however:

  1. How to link accounts with it? Assuming you'd want multiple API keys to each API user?
  2. Password to facilitate linking accounts?
  3. What if you lose your key?
  4. What about those mystical FPGA scrypt machines I've heard rumors about?

I might throw together an extremely simple "micro-blog" thing(twitter clone) that uses this concept just to see how it turns out. The hardest thing would probably be implementing scrypt in Javascript

Note One last thing. This isn't to "stop" spam. It's rather to make your site so expensive to spam that it's not profitable. Sure, you can always rent out a few hundred EC2 VMs or some such and compute a few hundred API tokens, but how much is that going to cost? How much do you expect to make from spamming that site?

Posted: 3/31/2013 4:11:10 AM

BarelyMVC now LucidMVC

So, finally decided to rename the project. However, I decided against ClearMVC as being a bit too plain. So, a close synonym to Clear is Lucid, hence LucidMVC. There is some small MVC PHP framework that was last updated last year that goes by the name of "lucid-php", but I bet I can beat his google page rank :)

Anyway, so there you have it. Update your repository URLs and such to use https://bitbucket.org/earlz/lucidmvc

Also, the BarelyMVC bitbucket repo still exists, but I had to create it from new. So, now it just has a single commit saying "this has been renamed"

Posted: 3/3/2013 6:40:21 PM

BarelyMVC meet ClearMVC

So, I think BarelyMVC isn't just a tiny minimalistic framework anymore. It's goals have changed. It's scope has changed. So, it's getting rebranded to ClearMVC. Hopefully, this will better reflect the primary goal of "no magical black box"

Posted: 2/19/2013 4:40:50 AM

Breaking Changes For Everyone!

So, remember how I said there would be no more breaking changes to the router of BarelyMVC? Well, part of the whole "making it testable" meant that the current API as it was sucked major balls. We need some way to simple get an IServerContext into the created HttpHandler. It's not really possible without magic with the current way the API is... So, it's changing.

The Proof Of Concept for a tiny taste of the new API is here. Highlights:

  • Fluent API blog.Handles("/blog/new").With((c)=> c.New()).RequiresAuthentication()
  • Worry less about getting data from routes/forms into your HttpHandler methods
  • Treat handlers more like controllers
  • Make it so no more reliance on static class elements like HttpContext.Current
  • Will reduce code duplication for adding similar routes on the same "controller"
  • STILL no reflection or manual casting required! Not even an explicit generic parameter!

With the way I foresee this working, I can honestly say it looks significantly better than ASP.Net MVC's way of routing. I mean, we're talking FLUENT API cool. I'd dare to say it's also better than OpenRasta's form of routing.

In case you were too lazy to look at that gist. Here is an example:

var blog=router.Controller(() => new BlogController());
blog.Handles("/foo/bar").With((c) => c.View());

Can it read anymore like plain English? I don't believe so. And still, no magic, no reflection, no casting. Just good ol' fashion generic delegates and some neat compiler support for implicit generic parameters.

So, yes, it's a huge breaking change, but your code will suck less after migrating. Trust me, I have about 50 lines of code just for routing for this blog. I don't take breaking changes to routing lightly.

Posted: 2/14/2013 7:17:37 AM

BarelyMVC Roadmap

So, I've been working on BarelyMVC recently and established that there isn't a formal roadmap. I think that's a bit of a disgrace and wish to change that. So, here is the road map target for version 1.0(in order sorta)

  1. Rework to use IServerContext so the entire framework is easily mocked and unit testable(and as a result, the application built on top of it) (note, API should be fairly stable throughout this conversion)
  2. Strive for better unit test coverage(Don't plan on measuring it, but a lot better than right now)
  3. Get session support built into FSCAuth
  4. Integrate CacheGen into BarelyMVC
  5. Documentation and a tutorial or two
  6. Visual Studio and/or MonoDevelop project templates
  7. Compare and contrast document between ASP.Net MVC and BarelyMVC
  8. Setup a CI and/or nightly build server
Posted: 1/20/2013 7:50:03 PM